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ABSTRACT 


The  thesis  identifies  F/A-18  squadron  characteristics  that  are  important 
predictors  of  maintenance  performance  and  draws  insights  on  the  linkage 
between  the  utilization  of  engineering  and  technical  services  (ETS)  and 
maintenance  performance  measures.  Statistical  analysis  is  conducted  to  identify 
squadron  characteristics  that  have  a  detectable  contribution  to  the  variability  of 
the  performance  measure  man-hours  per  maintenance  action,  and  how  much 
additional  variability  is  explained  by  the  squadron  that  is  not  accounted  for  by  the 
squadron  characteristics  already  considered. 

Thirty  months  of  data  were  collected  for  thirteen  active  duty  Marine  Corps 
F/A-18  squadrons.  Regression  is  used  to  model  man-hours  per  maintenance 
action  as  a  linear  combination  of  explanatory  variables  that  describe  the 
squadrons  in  terms  of  manpower,  inventory,  and  ETS  metrics.  The  test  for 
significance  indicates  that  the  model  developed  in  this  study  is  highly  likely  to 
have  better  explanatory  power  than  an  intercept-only  (average)  estimate  of  the 
response  variable.  The  study  concludes  with  recommendations  for  data 
collection  methods  that  would  facilitate  the  correlation  of  squadron  characteristics 
to  ETS  utilization.  Critical  to  the  success  of  this  approach  is  the  linkage  of  ETS 
utilization  to  specific  squadron  maintenance  activities,  and  the  development  of 
methods  to  quantify  maintainer  training  currency. 
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EXECUTIVE  SUMMARY 


The  performance  and  expertise  of  Naval  aviation  squadrons  is  closely  tied 
to  the  performance  of  their  maintenance  teams.  Aircraft  that  cannot  fly  or  operate 
in  a  fully  functional  manner  due  to  inadequate  maintenance  seriously  harms 
mission  capability.  It  is  useful,  therefore,  to  identify  factors  related  to  a 
squadron’s  mission,  and  the  personnel  and  assets  at  its  disposal,  which  help  to 
explain  the  performance  of  their  maintainers. 

How  should  maintainer  performance  be  measured?  The  speed  and 
correctness  with  which  maintenance  actions  are  conducted  are  important 
aspects  of  performance,  although  they  may  be  difficult  to  quantify.  External 
factors,  such  as  the  availability  of  repair  parts  and  the  operations  tempo  of  the 
squadron,  also  affect  measures  that  may  be  used  to  describe  maintenance 
performance.  Therefore,  we  use  man-hours  per  maintenance  action  as  a 
measure  of  performance,  due  to  its  direct  relationship  to  the  actions  of  the 
maintainers,  and  to  limits  the  effects  of  external  confounding  factors. 

In  this  thesis  we  examine  monthly  data  of  thirteen  Marine  Corps  F/A-18 
squadrons  taken  over  a  two-year  period  to  identify  squadron  characteristics  that 
are  important  predictors  of  man-hours  per  maintenance  action.  Also,  we  gain 
insight  on  maintenance  performance  from  data  collected  on  the  squadrons’ 
utilization  of  engineering  and  technical  services.  Specifically,  we  address  the 
following  research  questions: 

1.  Which  squadron  characteristics  have  a  detectable  contribution  to  the 
variability  of  the  performance  measure  man-hours  per  maintenance  action ? 

2.  How  much  additional  variability  is  explained  by  the  squadron  that  is  not 
accounted  for  by  the  squadron  characteristics  already  considered? 

3.  Is  there  a  time-of-year  effect  for  the  performance  of  the  squadrons? 

4.  What  additional  metrics  not  currently  available  would  most  likely  be 
useful  in  an  explanatory  model  of  maintenance  performance? 
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5.  What  data  collection  methods,  if  any,  would  be  likely  to  improve  the 
ability  of  NATEC  managers  to  correlate  squadron  characteristics  to  tech  rep 
measures  of  performance? 

Flight  operations  rely  on  a  maintenance  workforce  that  can  meet  the 
demands  of  a  flight  schedule  by  performing  preventive  and  corrective 
maintenance.  If  necessary,  maintenance  personnel  may  request  the  expertise 
offered  by  government  civil  service  or  civilian  contracted  personnel,  known  as 
technical  representatives  (“tech  reps”),  who  provide  engineering  and  technical 
services  (ETS)  in  the  form  of  on-the-job  training,  troubleshooting,  and  additional 
training. 

We  integrate  data  collected  from  several  independent  Department  of 
Defense  sources  on  maintenance  actions,  personnel,  aircraft  inventory,  and 
technical  services  utilization  to  derive  metrics  that  allow  performance  and  other 
characteristics  to  be  quantified  for  the  thirteen  Marine  Corps  F/A-18  squadrons  in 
the  scope  of  our  study.  For  each  squadron,  approximately  30  months  of 
observations  are  collected  to  quantify  performance  and  descriptive 
characteristics.  Personnel  metrics  quantify  the  experience  levels  and  turnover 
rates  of  the  squadrons  on  a  monthly  basis.  Experience  is  measured  by  the 
number  of  months  that  an  individual  maintainer  has  spent  in  a  squadron  and  in 
the  Marine  Corps.  Inventory  metrics  characterize  the  ages  and  type  of  F/A-18 
aircraft  maintained  by  a  squadron.  Technical  services  metrics  quantify  the  type 
and  volume  of  ETS  activity  in  a  squadron  for  a  given  month.  We  also  capture  the 
operational  context  in  which  a  squadron  performs  its  mission:  combat  operations, 
unit  deployment  program,  a  carrier  deployment,  or  a  between-deployment  phase. 

Exploratory  data  analysis  shows  that  performance  cannot  be  explained  by 
any  single  squadron  characteristic.  Linear  regression  is  used  to  model  man¬ 
hours  per  maintenance  action  as  a  linear  combination  of  explanatory  variables.  A 
test  for  significance  indicates  that  the  model  is  highly  likely  to  explain  the 
variability  of  the  response  variable  when  compared  to  an  intercept-only  (average 
response)  model.  Stepwise  reduction  is  used  to  reduce  the  model  to  a  simpler 
model  that  retains  most  of  its  explanatory  power.  This  reduced  model  indicates 


that  five  of  the  explanatory  variables  are  statistically  significant  in  explaining  man¬ 
hours  per  maintenance  action',  type  equipment  code  (TEC),  average  aircraft 
hours  in  service,  median  months  in  squadron,  location,  and  deployment  status. 
Nonetheless,  this  model  explains  only  20  percent  of  the  variability  of  the 
response  variable.  By  including  a  factor  that  identifies  the  particular  squadron, 
the  explanatory  capability  of  the  model  is  increased  to  approximately  50  percent. 
This  suggests  that  there  are  important  differences  between  the  squadrons  that 
explain  performance  but  that  are  not  captured  in  the  variables  included  in  this 
study.  The  final  model  takes  the  form 


ln  Ys,t  -  A)  +  P\X\ ,s,t  +  PlX2 ,s,t  +  PiX2,s,t  +  f^4X4,s,t  + 

PsX5  ,sJ  +  £s.t 


Where 


Ys  t  =  man-hours  per  maintenance  action,  squadron  s,  month  t 
Xx  =  type  equipment  code 
X2  s  (  =  average  aircraft  hours  in  service 
X3st  =  location 

X4  s  t  =  months  in  squadron,  median 
X5  s  t  =  deployment  status 
sst  =  residual 
k  =  number  of  variables 
s  =  squadron 
t  =  month 


For  those  factors  found  to  be  significant,  the  coefficients  provide  some 
insight  as  to  their  positive  or  negative  correlation  with  the  performance  variable. 
The  performance  of  the  maintainers,  expressed  as  man-hours  per  maintenance 
action,  improves  (decreases)  with  increased  experience  of  the  maintainers 
(higher  values  of  months  in  squadron). 

The  data  do  not  indicate  that  there  is  a  time-of-year  effect  in  man-hours 
per  maintenance  action.  However,  there  is  detectable  serial  correlation  in  the 
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residuals  from  the  regression  model,  suggesting  that  there  may  be  temporal 
effects  that  could  be  handled  with  a  generalized  least  squares  approach. 

The  thesis  is  constrained  primarily  by  the  short  time  frame  of  the  study,  a 
result  of  the  attempt  to  include  the  relatively  recent  ETS  data  in  the  analysis.  At 
the  time  of  this  writing,  ELAR  is  a  nascent  database  with  records  of  varying 
degrees  of  completeness.  In  a  broader  sense,  NATEC’s  ELAR  initiative  and  this 
thesis  are  both  part  of  a  larger  effort  to  link  maintenance  utilization  metrics,  one 
of  which  is  ETS  utilization,  with  maintenance  performance  measures.  As  the 
quality  and  scope  of  ELAR  data  reporting  improve,  ELAR  will  play  a  more 
effective  role  in  establishing  a  linkage  between  ETS  utilization  and  Naval  aviation 
maintenance  performance.  In  addition,  the  explanatory  power  of  the  model  would 
likely  improve  with  more  accurate  model  estimates  obtained  from  data  collected 
over  a  longer  period  of  time,  and  from  the  inclusion  of  maintenance  performance 
metrics  not  currently  available  in  the  maintenance  data  system. 

The  study  concludes  with  recommendations  for  data  improvement.  We 
determine  that  a  critical  requirement  for  making  the  tech  rep  data  more  valuable 
to  analysis  is  the  linking  of  their  activity  to  specific  squadron  maintenance 
activity — through  NALCOMIS,  for  example.  This  will  allow  a  direct  measure  of 
their  impact  on  readiness  and  performance  in  a  way  similar  to  other  maintenance 
factors. 

Also  vital  to  the  description  of  squadron  capability  is  the  development  of 
methods  to  quantify  the  training  currency  of  the  maintainers.  This  will  allow  real¬ 
time  assessment  of  maintenance  proficiency  and  will  highlight  skill  areas  that 
need  renewed  training  attention. 


ACKNOWLEDGMENTS 


The  successful  completion  of  this  thesis  was  ensured  by  the  generous 
contributions  and  assistance  of  many.  Critical  to  success  were  those  individuals 
that  provided  guidance  through  the  labyrinth  of  data  needed  for  this  analysis. 
Chris  Hawes  of  NAVAIR  responded  immediately  and  frequently  to  the  endless 
stream  of  questions  for  clarification  and  explanation.  LtCol  Marcus  Messina, 
USMC,  of  Defense  Manpower  Data  Center  diverted  resources  to  answer  my 
unusual  data  requests.  Several  individuals  at  NATEC  guided  the  project  and 
provided  assistance  from  beginning  to  end;  NATEC  Commanding  Officer 
Commander  Joe  Beel,  USN,  Mike  Zabrouskas,  and  Bob  Anderson  are 
noteworthy.  Bob  Kaitchuk  from  Anteon  provided  aircraft  inventory  data  from 
independent  sources  that  would  have  been  impossible  to  compile  on  my  own.  I 
am  grateful  to  the  many  more  unnamed  individuals  of  the  operating  forces. 

Further  thanks  must  go  to  the  faculty  of  the  Operations  Research 
department  at  the  Naval  Postgraduate  School — a  truly  dedicated  group  of 
professionals  who  apply  their  expertise  to  real  issues  concerning  the  military 
services.  Specifically,  though,  I  have  to  thank  the  analysts  that  made  the 
educational  experience  truly  enjoyable:  Professors  Sam  Buttrey,  Lyn  Whitaker, 
and  Robert  Read.  Their  ability  to  share  knowledge  is  invaluable.  The  top  of  the 
list  must  be  reserved  for  my  thesis  advisor,  Professor  Robert  Koyak,  whose 
approach  to  problem-solving  was  inspiring  and  whose  patience  was  unlimited. 

In  the  end,  I  am  indebted  to  my  wife  for  her  reassuring  support  and 
encouragement  through  this  endeavor. 


XIX 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


xx 


I.  INTRODUCTION 


When  you  can  measure  what  you  are  speaking  about,  and  express 
it  in  numbers,  you  know  something  about  it;  but  when  you  cannot 
measure  it,  when  you  cannot  express  it  in  numbers,  your 
knowledge  is  of  a  meager  and  unsatisfactory  kind:  it  may  be  the 
beginning  of  knowledge,  but  you  have  scarcely,  in  your  thoughts, 
advanced  to  the  stage  of  science,  what  ever  the  matter  may  be. 

Lord  Kelvin  1824-1907,  British  scientist 

A.  BACKGROUND 

1 .  The  Value  of  Human  Capital 

We  have  evolved  from  an  industrial  age  during  which  assets  were 
tangible,  countable,  and  of  a  measurable  value,  to  a  modern  era,  characterized 
by  the  need  for  and  availability  of  information.  Today’s  organizations  are  unique 
in  that  a  large  percentage  of  their  worth  lies  in  the  value  of  their  human  capital 
rather  than  its  physical  assets.  In  the  past  three  decades,  both  public  and  private 
industry  has  been  forced  to  streamline  operations  in  the  face  of  reduced  budgets, 
smaller  margins,  and  focused  competition.  At  the  heart  of  this  struggle  is  the 
need  to  measure  the  value  of  that  human  capital — its  output  and  its  capacity  to 
produce. 

This  problem  is  not  confined  to  the  corporate  world  of  the  balance  sheet 
and  the  profit  and  loss  statement;  with  limited  resources,  the  U.S.  Department  of 
Defense  also  needs  to  maximize  the  output  of  its  personnel  and  equipment.  The 
services  have  seen  an  additional  focus  on  force  transformation  [Rumsfeld,  2003]; 
such  a  level  of  productivity  demands  streamlined,  optimized  operations.  In  the 
face  of  these  requirements,  leaders  strive  to  develop  metrics  that  will  enable 
them  to  accurately  measure  productivity  and  the  factors  that  improve  it. 

2.  Aviation  Maintenance 

A  functional  area  under  continued  scrutiny  in  both  the  civilian  and  military 
sectors  is  that  of  aviation  maintenance.  Although  their  missions  are  different,  as 
are  the  environments  in  which  their  missions  are  executed,  both  military  and 
civilian  flight  operations  rely  on  a  maintenance  workforce  that  can  meet  the 
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demands  of  a  flight  schedule.  Civilian  organizations  may  see  sub-optimal 
performance  reflected  in  reduced  profit.  The  armed  services,  on  the  other  hand, 
may  not  see  immediate  ramifications  of  poor  performance.  The  ultimate  test  for 
any  military  unit  is  combat,  but  with  sporadic  combat  operations,  the 
consequences  of  inferior  maintenance  performance  are  not  always  apparent. 

An  understanding  of  the  functions  of  aviation  maintenance  helps  to  explain 
how  analysts  attempt  to  measure  performance.  In  the  most  basic  terms,  the 
mission  of  any  aviation  maintenance  organization  is  straightforward:  to  maintain 
aircraft  through  routine  scheduled  maintenance  and  to  repair  aircraft  that  become 
inoperable  due  to  normal  use  and  wear.  Aviation  maintenance  managers  must 
strike  the  proper  balance  between  scheduled  and  unscheduled  maintenance  to 
meet  the  demands  of  a  squadron’s  flight  schedule  while  preserving  the  long-term 
health  of  the  fleet. 

All  military  flying  units  have  the  ability  to  perform  a  limited  level  of 
maintenance  on  their  own  inventory  of  aircraft.  During  the  course  of  an  operating 
day,  any  aircraft  malfunctions  that  are  not  discovered  by  the  maintainers  through 
routine  inspection  are  usually  brought  to  their  attention  by  the  aircrew  that 
discover  them  either  before,  during,  or  after  a  flight.  The  aircrew  and 
maintenance  personnel  record  these  discrepancies  electronically,  which  initiates 
the  maintenance  process  required  to  address  the  discrepancy.  The  discrepancy 
record  also  contains  the  repair  time,  man-hours  expended,  parts  removed  and 
replaced,  and  other  descriptive  information.  The  purpose  of  such  data  collection 
is  to  allow  maintenance  analysts  to  identify  trends  that  may  point  to  problem 
areas  such  as  high-fault  subsystems  and  repeat  discrepancies. 

3.  Engineering  and  Technical  Services 

Throughout  the  course  of  maintenance  being  performed  on  the  aircraft — a 

process  that  ends  with  the  action  being  approved  by  a  responsible  authority — 

maintenance  personnel  (“maintainers”)  diagnose  the  discrepancy  by  referencing 

their  own  training  and  experience,  technical  publications  with  prescribed 

troubleshooting  techniques,  and  other  personnel  who  may  have  performed 

similar  maintenance  in  the  past.  If  necessary,  maintainers  may  request  the 
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expertise  offered  by  government  civil  service  or  civilian  contracted  personnel  who 
provide  engineering  and  technical  services.  In  addition  to  providing  on-site 
troubleshooting  expertise,  these  service  providers,  referred  to  as  “tech  reps”  for 
short,  supplement  the  training  of  maintenance  personnel  by  providing  more 
formal  instruction  in  classroom  settings  and  in  squadron  work  centers. 

In  the  Department  of  the  Navy,  tech  reps  are  managed  by  the  Naval  Air 
Technical  Data  and  Engineering  Service  Command  (NATEC).  The  origin  of  what 
is  now  NATEC — formerly  known  as  NAESU  (Naval  Aviation  Engineering  Service 
Unit) — was  the  response,  in  WWII,  to  the  shortage  of  trained  electronics 
technicians.  Now  responsible  for  all  areas  of  engineering  and  technical  data, 
NATEC  documents  requests  for  assistance  in  a  database  called  ELAR  (ETS 
Local  Assistance  Request).  ELAR  records  are  generally  initiated  by  the 
maintenance  activity  that  requests  NATEC  support.  The  requests  are  approved 
and  apportioned  by  a  NATEC  detachment  supervisor,  and  are  finalized  with  brief 
customer  satisfaction  comments  from  the  originator  upon  completion  of  the 
action. 

NATEC’s  customers — the  flying  squadrons  of  the  Navy  and  Marine 

Corps — have  grown  accustomed  to  having  the  availability  of  the  tech  reps  at  their 

disposal  even  during  operational  deployments.  However,  NATEC  must  allocate 

its  limited  resources  to  meet  the  competing  demands  of  its  customers.  Such  an 

allocation  involves  determining  the  best  performance  value  return  on  manpower 

resource  investment.  NATEC  managers,  like  other  maintenance  managers,  seek 

to  define  those  metrics  that  best  measure  the  health  of  the  squadrons  in  order  to 

optimize  the  distribution  of  their  limited  resources  and  maximize  customer 

satisfaction.  Analyses  of  tech  rep  support  [Boynton,  Seiden,  and  Vaughan  1995; 

Boynton  and  Vaughan,  1998]  describe  the  difficulties  of  quantifying  tech  reps’ 

contributions  to  aviation  readiness.  NATEC  implemented  ELAR  in  August  2003  in 

an  effort  to  address  this  problem.  Prior  to  ELAR,  the  Navy  lacked  a  systematic 

data-collection  tool  for  tracking  the  utilization  of  technical  services  by  its  aviation 

maintainers.  In  the  absence  of  such  data  it  is  impossible  to  correlate  aviation 

maintainer  performance  to  the  usage  of  these  services.  At  the  time  of  this  writing, 

3 


ELAR  contains  approximately  two  years  of  data,  but  its  early  records  are 
insufficiently  complete  to  conduct  meaningful  statistical  analyses  linking  ETS 
utilization  to  maintenance  outcomes.  We  view  the  continual  improvement  of 
ELAR  and  this  thesis  as  parts  of  a  larger  effort  to  link  characteristics  of  aviation 
maintainer  communities,  including  their  ETS  utilization,  to  maintenance 
performance  measures.  We  expect  that  as  the  quantity  and  quality  of  ELAR  data 
continues  to  increase,  greater  success  in  this  endeavor  will  be  realized. 

B.  LITERATURE  REVIEW 

We  proceed  with  a  review  of  literature  that  addresses  the  analysis  of 
aviation  maintenance  performance.  Of  particular  interest  to  us  are  studies  that 
measure  the  contribution  of  maintainers  to  the  performance  of  U.S.  military 
aviation  fighting  units.  We  begin  our  review  with  research  concerned  with 
measuring  performance.  We  then  address  studies  supported  by  the  United 
States  Air  Force,  an  organization  that  faces  maintenance  performance  issues 
similar  to  those  of  the  Naval  Aviation  community.  Finally,  we  address  studies  on 
the  effectiveness  of  engineering  and  technical  services. 

1.  Measuring  Performance 

One  can  quantify  the  accomplishments  of  aviation  units  in  many  ways: 
missions  flown,  targets  struck,  aircraft  repaired,  etc.  Some  of  these  measures  are 
operational  in  nature,  indicating  the  performance  of  the  aircrew  and  their  level  of 
training,  while  others  focus  on  the  performance  of  maintenance  personnel.  Data 
elements  are  captured  by  both  aircrew  and  technicians  during  and  after  each 
flight  event  or  maintenance  action,  allowing  analysts  to  calculate  metrics  that 
describe  the  output  of  the  unit’s  maintenance  effort.  Commanders  are  also 
interested  in  their  unit’s  ability  to  accomplish  future  missions.  To  this  end,  the 
Defense  Department  and  the  Services  adopted  metrics  to  quantify  unit 
readiness.  The  Department  of  Defense  Dictionary  of  Military  and  Associated 
Terms  (Joint  Publication  1-02)  [CJCS,  2001]  defines  readiness  as  follows: 
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Readiness.  The  ability  of  US  military  forces  to  fight  and  meet  the 
demands  of  the  national  military  strategy.  Readiness  is  the 
synthesis  of  two  distinct  but  interrelated  levels,  a.  unit  readiness  — 

The  ability  to  provide  capabilities  required  by  the  combatant 
commanders  to  execute  their  assigned  missions.  This  is  derived 
from  the  ability  of  each  unit  to  deliver  the  outputs  for  which  it  was 
designed  b.  joint  readiness  —  The  combatant  commander’s  ability 
to  integrate  and  synchronize  ready  combat  and  support  forces  to 
execute  his  or  her  assigned  missions.  [CJCS,  2001 ,  p.  440], 

The  Training  and  Readiness  Manual  (T&R  Manual)  describes  the 
readiness  models  that  standardize  training  and  readiness  methodology.  Navy 
and  Marine  Corps  aviation  decision-makers  use  these  models  to  plan  and  budget 
for  the  appropriate  number  of  sorties  and  flight  hours  to  support  unit  readiness 
goals,  which  in  turn  places  demands  on  resource  (aircraft)  readiness.  At  the  unit 
level,  commanders  and  their  operational  staffs  use  aircraft  to  appropriately  meet 
training  requirements. 

a.  Mission  Capable  (MC)  Rates 

Aviation  maintenance  analysts  often  focus  on  aircraft  readiness 
rates  as  a  primary  maintenance  performance  indicator.  Readiness  is  measured 
as  an  overall  mission  capable  (MC)  rate,  or  as  not  mission  capable  (NMC)  or 
partially  mission  capable  (PMC)  rates.  Mission  capability  is  adversely  impacted 
when  a  system  or  subsystem  renders  an  aircraft  incapable  of  performing  its 
missions,  as  when  components  are  removed  from  an  aircraft  for  repair  or 
replacement.  System  failures  that  deny  mission  capability  are  codified  in  the 
Mission-Essential  Subsystem  Matrices  (MESM),  which  are  available  to  the 
maintenance  crews  of  Naval  aircraft.  The  MESM  for  F/A-18  subsystems  can  be 
found  in  Appendix  A. 

As  noted  above,  mission  capability  can  be  delineated  in  various 
ways.  Fully  mission  capable  (FMC)  signifies  that  an  aircraft  can  perform  all  of  its 
missions.  Partially  mission  capable  (PMC)  indicates  that  an  aircraft  can  perform 
one  or  more — but  not  all — of  its  assigned  missions.  Not  mission  capable  (NMC) 
implies  that  an  aircraft  can  perform  none  of  its  missions,  which  may  be  further 
delineated  as  not  mission  capable  due  to  maintenance  (NMCM)  and  not  mission 
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capable  due  to  supply  (NMCS).  Not  mission  capable  due  to  maintenance 
(NMCM)  is  then  distinguished  as  being  due  to  scheduled  (NMCMS)  or 
unscheduled  (NMCMU)  maintenance.  Each  of  these  aspects  of  mission 
capability  is  used  to  describe  a  particular  aircraft  attached  to  a  squadron,  which  is 
then  aggregated  across  time  to  produce  monthly  aircraft  or  unit-based  rates.  For 
example,  an  aircraft’s  accrued  MC  time  is  the  total  time  that  the  aircraft  is  in 
service  less  its  NMC  time.  Collectively,  these  metrics  are  monitored  in  the 
Subsystem  Capability  and  Impact  Reporting  (SCIR)  system,  which  in  turn 
provides  data  to  the  Maintenance  Data  System  (MDS).  The  reader  is  referred  to 
OPNAVINST  5442. 4M  [Chief  of  Naval  Operations,  1990]  for  a  detailed 
discussion  of  SCIR  and  MDS. 

The  MESM  is  limited  to  subsystems  that  are  most  likely  to  affect 
mission  success  and  aircrew  safety.  As  an  aircraft  is  modified  through  the 
addition  of  more  complex  weapons,  software,  avionics,  and  even  missions, 
commanders  and  maintenance  managers  must  often  deal  with  critical 
subsystems  that  are  not  explicitly  listed  in  the  MESM,  which  is  over  a  decade  old 
at  the  time  that  this  thesis  is  written.  In  the  absence  of  common  standards  of 
interpretation,  individual  units  may  introduce  variability  in  the  classification  of 
mission  capability  status. 

MC  rates  are  subject  to  close  scrutiny  at  all  levels  of  command.  In 
their  testimony  before  the  U.S.  House  of  Representatives  Committee  on  National 
Security,  Steele  and  Dake  [1998a]  use  MC  rates  to  chronicle  an  eight-year 
decline  in  the  readiness  of  Marine  Corps  warfighting  units.  They  attribute  this 
decline  to  the  aging  of  the  services’  aircraft  and  the  corresponding  decrease  in 
reliability. 

b.  Supply  Indicators 

Steele  and  Dake  [1998a]  identify  a  category  of  replacement  parts, 

shortages  of  which  lead  to  high  rates  of  cannibalization  and,  consequently, 

increased  maintenance  workload,  as  evidenced  in  overlapping  and  rotating 

shifts.  We  infer  from  their  reference  to  the  increased  maintenance  workload  the 

importance  of  metrics  that  quantify  man-hours  required  to  produce  repairs  and 
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subsequent  sorties.  We  discuss  these  in  detail  in  Chapter  II.  Maintenance 
analysts  monitor  the  NMCS  rate  to  gauge  the  effects  of  the  supply  system  on 
readiness.  During  a  maintenance  action,  the  aircraft  will  go  through  stages  of 
repair  that  involve  Awaiting  Maintenance  (AWM),  Elapsed  Maintenance  Time 
(EMT),  and  Awaiting  Parts  (AWP).  The  time  accrued  in  AWP  status  is  used  as 
another  gauge  of  the  supply  system. 

c.  Readiness  Management  Tools 

The  Global  Status  of  Resources  and  Training  System  (GSORTS)  is 
a  data  base  that  was  designed  to  provide  the  service  branches,  unified 
commands,  and  combat  support  agencies  with  the  ability  to  monitor  readiness  of 
warfighting  units.  GSORTS  provides  the  levels  of  selected  resources  and  training 
required  to  undertake  the  mission(s)  for  which  a  unit  is  responsible.  GSORTS 
consists  of  classified  data  entered  by  every  operational  unit,  which  they  submit  at 
monthly  intervals  or  when  dictated  by  other  milestones  such  as  unit  deployments. 
Senior  military  officials  monitor  GSORTS  reports  to  detect  deviations  from 
desired  readiness  trends.  In  their  quarterly  readiness  report  to  Congress,  Steele 
and  Dake  [1998b]  explain  how  the  Marine  Corps  uses  GSORTS  as  a  tool  for 
monitoring  readiness.  They  also  note  that  readiness  rates  should  be  viewed  in 
the  proper  context,  since  units  operate  on  a  cycle  of  readiness  that  accepts  lower 
rates  during  post-deployment  times  when  priorities  shift  to  deploying  units: 

If  a  unit  is  reporting  low  readiness,  we  first  compare  the  unit’s 
status  in  the  cyclical  deployment  queue  or,  in  the  case  of  a 
detachment-providing  unit,  the  number  of  detachments  currently 
deployed.  If  these  first-order  cuts  provide  no  illumination  of  the 
problem,  the  next  step  is  to  compare  current  reporting  against 
historical  trends  [Steele  and  Dake,  1998b,  p.3]. 

The  GSORTS  system  categorizes  readiness  with  respect  to  the 
following:  equipment  on  hand,  equipment  condition,  personnel,  and  training.  The 
reporting  unit  assigns  to  each  of  these  categories  a  rating  that  is  derived  from  the 
measures  of  effectiveness  for  that  category.  The  unit  commander  also  provides  a 
subjective  overall  rating  of  unit  readiness  known  as  the  C-rating,  in  order  to 
address  intangibles  not  captured  in  numerical  data. 
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Orlansky,  Hammon,  and  Horowitz  [1997]  seek  to  identify  reliable 
indicators  of  exercise  and  combat  performance.  Specifically,  they  analyze  the 
ability  of  GSORTS  metrics  to  accurately  predict  performance: 

Service  ...  indicators  include:  personnel,  training,  equipment, 
supply,  operating  tempo,  commitments  and  deployments,  funding, 
and  accident  rates.  Although  these  indicators  appear  likely,  at  least 
intuitively,  to  influence  readiness,  no  analysis  was  provided  to  show 
that  variations  in  any  of  them  are  consistently  related  to  variations 
in  readiness.  Such  work  needs  to  be  done  before  one  should 
conclude  that  adding  any  of  these  indicators  would  improve  our 
ability  to  evaluate  current  or  predict  future  readiness.  [Orlansky, 

1997,  p.  S-3] 

The  authors  examine  the  performance  metrics  used  by  the 
Services  and  compare  them  with  measurable  results  from  large-scale  exercises, 
readiness  evaluations,  and  combat.  They  find  positive  correlation  between  many 
of  the  currently  used  measures  of  training  readiness  and  unit  performance. 
Specifically,  they  recommend  the  use  of  those  metrics  shown  in  Figure  1  : 
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Figure  1  Potential  Training  Readiness  Indicators  [Orlansky,  1997,  p.  IV-2]. 
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In  addition  to  GSORTS,  the  authors  consider  several  other  sources 
of  readiness  information  in  their  analysis,  such  as  the  Type  Commander 
Readiness  Management  System  (TRMS),  which  is  a  task-based  readiness 
management  system.  At  the  time  of  this  writing,  TRMS  does  not  incorporate 
measures  of  aviation  maintenance  training  readiness. 

Orlansky,  et  al.  recommend  that  the  following  data-driven  activity 
be  undertaken  to  define  the  linkages  between  readiness  and  factors  that  may 
drive  readiness: 

1 .  Analyze  (readiness)  data  to  identify  short  term  and  long  term 
trends,  including  noise;  i.e.,  short-term,  non-significant 
variations. 

2.  Where  trends  are  observed,  identify  the  time  delays  between 
inputs,  i.e.,  resources,  process,  and  outputs — the  related 
consequences  in  ...demonstrated  combat  capability. 

3.  Examine  indicators  for  redundancy  [that]  add  little  additional 
information  about  status  and  trends. 

4.  Examine  indicators  that  could  be  combined  by  appropriate 
statistical  procedures. 

5.  Examine  the  relation  between  subjective  and  objective 
indicators  of  readiness. 

6.  Start  the  collection  and  analysis  of  new  demonstrated 
performance  measures.  [Orlansky,  1997,  p.  V-2], 

The  authors’  recommendations  highlight  specific  areas  that  should 

be  considered  in  cause-effect  studies  and  that  we  address  in  our  analysis: 

unexplained  variance,  lag  effects,  and  multicollinearity,  the  interdependence 

among  explanatory  variables. 

Orlansky,  et  al.  cite  the  findings  of  Junor  and  Oi  [1996],  who 
examine  all  areas  of  GSORTS  readiness,  in  addition  to  training.  Junor  and  Oi 
identify  27  metrics,  used  by  the  Navy  surface  warfare  community,  that  are 
effective  in  explaining  or  forecasting  GSORTS  readiness  levels.  In  related  work, 
Robinson,  Jondrow,  Junor,  and  Oi  [1996]  identify  trends  in  Navy  readiness.  They 
use  cluster  analysis  to  divide  time-series  observations  into  discrete  time  intervals 
with  minimum  variability,  and  then  employ  principal  components  to  describe  the 
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relationship  between  these  clusters  as  an  indication  of  trend.  They  state  in  their 
findings  that  readiness  tends  to  move  in  long  slow  cycles  and  that  GSORTS  is  a 
useful  measure  of  readiness. 

Although  the  studies  described  above  do  not  address  aviation 
maintenance  performance  metrics  specifically,  they  provide  insight  into 
approaches  and  methodologies  that  prove  useful  in  our  analysis. 

2.  U.S.  Air  Force  Studies  of  Maintenance  Performance 

The  Air  Force  has  devoted  considerable  effort  to  measuring  maintenance 
performance  and  its  contributing  factors.  Oliver  [2001]  examines  the  readiness 
data  of  U.S.  Air  Force  F-16C/D  aircraft  over  a  ten-year  period  with  the  goal  of 
identifying  those  factors  that  contribute  to  readiness.  He  stresses  that  current  Air 
Force  readiness  forecasting  models,  while  accurate,  are  predictive  rather  than 
explanatory  models.  He  compiles  nine  years  of  data  from  aviation  maintenance, 
personnel,  and  logistics  sources,  such  as  the  Reliability  and  Maintainability 
Information  System  (REMIS),  the  Personnel  Data  System  (PDS),  and  Manpower 
Data  System  (MDS),  to  derive  606  variables  that  may  explain  MC  rate  variability. 
Additionally,  Oliver  considers  versions  of  each  variable  that  are  lagged  by  one, 
two,  three,  and  four  quarters,  respectively,  resulting  in  a  total  of  3030  variables. 
He  then  reduces  the  set  of  variables  by  eliminating  those  with  low  correlation  to 
the  MC  rate,  and  redundant  variables  that  contribute  to  multicollinearity.  After 
setting  aside  eight  quarters  of  data  as  a  test  set,  he  employs  linear  regression 
and  stepwise  regression  to  identify  the  smallest  significant  model  that  explains 
MC  rates  for  Air  Force  F-16  squadrons.  His  final  step  is  to  build  a  predictive 
model  for  MC  rate  that  includes  only  those  variables  that  can  be  controlled  by 
decision-makers. 
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Oliver  recognizes  that  readiness  is  a  complex  phenomenon  affected  by 
many  input  variables: 

Most  of  the  variables  contained  within  each  area  are  interrelated 
with  one  another  so  that  changes  in  one  variable  may  cause  a 
“ripple  effect”  that  impacts  other  variables  [and]  the  research 
indicated  that  unforeseen  changes  in  the  world  environment 
(environmental  variables)  created  a  series  of  powerful  “ripple 
effects”  which  lead  to  a  series  of  decisions  that  significantly 
influenced  mission  capable  rates.  [Oliver,  2001,  p.  110] 

Oliver’s  study  covers  the  ten-year  period  during  which  U.S.  military  fighting 
forces  experienced  a  substantial  decline  in  readiness,  in  large  part  due  to 
downsizing  after  the  collapse  of  the  Soviet  Union  signaled  the  end  of  the  Cold 
War.  In  his  models  Oliver  attempts  to  control  for  this  historical  trend  in  order  to 
isolate  the  effect  on  readiness  due  to  the  manning  and  experience  levels  of  F-16 
maintainers.  He  finds  that  manning  (expressed  as  maintainers  per  aircraft)  and 
experience  (expressed  as  rank,  skill  level,  or  job  assignment)  are  highly 
significant  in  explaining  readiness  in  terms  of  MC  rates. 

In  December  2001,  the  Air  Force  Logistics  Management  Agency  (AFLMA) 
published  The  Metrics  Handbook  for  Maintenance  Leaders  [AFLMA,  2001],  In 
this  document,  Air  Force  aircraft  maintenance  metrics  are  standardized,  defined, 
and  their  importance  explained.  This  handbook  categorizes  metrics  as  leading  or 
lagging  according  to  whether  the  effects  that  they  measure  occur  early  or  late  in 
the  causal  chain: 

Leading  indicators  are  those  that  directly  impact  maintenance’s 
capability  to  provide  resources  to  execute  the  mission.  Lagging 
indicators  show  firmly  established  trends.  In  other  words,  the 
leading  indicators  will  show  a  problem  first,  and  the  lagging 
indicators  will  follow.  [AFLMA,  2001,  p.  14] 

MC  rate  is  a  lagging  metric,  as  defined  by  AFLMA.  Choosing  MC  rate  as  a 
response  variable,  as  does  Oliver  [2001],  calls  for  the  consideration  of  many 
explanatory  variables  that  precede  the  manifestation  of  the  lagging  metric. 
Leading  metrics,  such  as  repeat  and  recurrence  (R/R)  rates  and  8-hour  fix  rates, 
however,  more  immediately  reflect  the  effects  of  the  input  variables.  In  addition  to 
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defining  performance  metrics  for  aviation  maintenance,  the  handbook  provides 
insights  into  their  effective  use  in  the  identification  of  trends,  diagnosis  of 
problems,  and  inclusion  in  narratives  directed  towards  decision-makers. 

Beabout  [2003],  who  develops  a  visual  tool  to  identify  troublesome  aircraft 
subsystems,  also  operates  within  the  context  of  leading  and  lagging  indicators. 
He  offers  techniques  for  separating  scheduled  maintenance  from  unscheduled 
maintenance  activity  to  isolate  those  subsystems  that  cause  high  NMC  rates. 

In  RAND’s  Project  Air  Force  case  study  of  an  Air  Force  Fighter  Wing, 
Dahlman  and  Thaler  [2000]  consider  how  imbalances  in  manning  lead  to 
shortfalls  in  readiness.  The  authors  begin  by  describing  two  competing 
foundations  of  readiness:  a  unit’s  ability  to  respond  to  current  operational 
demands  of  the  combatant  commanders  and  its  ability  to  produce  future 
capabilities  through  the  rejuvenation  of  human  capital: 

As  units  are  deployed  to  support  contingency  operations,  they  must 
trade  off  building  future  capabilities  for  providing  current  ones.  The 
longer  this  continues,  the  more  units  must  postpone  or  scale  back 
upgrade  training  and  life-cycle  maintenance  of  aircraft.  Future 
commanders  then  have  a  less  experienced,  less  capable  force  from 
which  to  draw.  [Dahlman  and  Thaler,  2000,  p.  2] 

By  acknowledging  such  competing  requirements  and  the  constant  loss  of 
qualified  personnel  from  the  unit,  Project  Air  Force  analysts  define  a  ‘healthy’ 
squadron  in  terms  of  its  appropriate  distribution  of  manpower  across  all  skill 
levels.  Specifically,  they  recommend  a  mix  of  maintainer  experience  that  will 
provide  adequate  on-the-job  training  (OJT)  over  time. 

The  problem  is  made  evident  by  drawing  analogies  to  the  more  familiar 
phenomenon  occurring  in  the  area  of  aircrew  training,  which  ordinarily  is 
conducted  in  an  environment  in  which  the  number  of  sorties  is  constrained  by 
utilization  rate  limits  or  by  budget.  As  inexperienced  pilots  join  the  unit,  and 
experienced  pilots  are  transferred  to  other  assignments,  an  increased 
percentage  of  the  fixed  number  of  overall  available  sorties  must  be  flown  by 
instructor  pilots,  leaving  a  smaller  percentage  of  sorties  for  the  junior  aircrew. 
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The  effect  compounds  over  time,  since  it  takes  longer  for  those  junior  aircrew  to 
meet  the  minimum  requirements  to  be  considered  combat  proficient.  Dahlman 
and  Thaler  recognize  the  parallel  problem  occurring  on  the  maintenance  side: 


The  dilemma  emerges  when  experienced  personnel  leave  at  a 
faster  rate  than  junior  personnel  can  be  adequately  trained  and 
promoted.  ...  The  USAF’s  response  to  diminishing  retention  rates 
largely  has  been  to  push  more  new  personnel  into  critical  career 
fields  that  are  losing  experienced  personnel.  This  presents  the  wing 
with  a  “Catch-22” — it  is  losing  experienced,  productive 
maintainers/trainers  and  gaining  inexperienced  3-  level  trainees, 
who  require  more  of  the  experienced  maintainers/trainers,  whom  it 
can  generally  gain  only  by  training  3-levels.  In  the  extreme,  the 
additional  workload  can  exacerbate  the  exodus  of  experienced 
personnel  from  the  force,  further  compounding  the  problem. 
[Dahlman  and  Thayer,  2000,  p.  13] 

Other  metrics  gathered  to  support  this  hypothesis  come  from  surveys  of 
maintainers.  These  surveys  indicate  that  senior  maintainers  spend  a  large 
portion  of  their  workday  on  repair  activities,  as  opposed  to  training  or  supervisory 
activities.  Also  significant,  according  to  their  data,  is  the  increased  time  required 
for  advancement  from  low  (“3-level”)  to  medium  (“5-level”)  skill  levels.  The  Air 
Force  uses  “3-level,”  “5-level,”  and  “7-level”  designations  as  indicators  of 
experience. 

There  is  a  symbiotic  relationship  between  the  maintenance  and 
operational  sides  of  an  aviation  fighting  unit:  each  must  work  to  the  benefit  of  the 
other.  The  aircrew  need  higher  utilization  rates  from  the  aircraft  to  achieve  the 
higher  number  of  sorties  required  to  train  the  less  experienced  aircrew;  however, 
higher  utilization  rates  on  the  aircraft  leave  less  time  for  maintaining  aircraft  and 
for  training  junior  maintainers,  reinforcing  the  problems  already  faced  by  the 
maintenance  community.  Dahlman  and  Thaler  [2000]  explain  how  an  imbalance 
in  this  relationship  ultimately  is  felt  in  readiness: 

In  sum,  our  analysis  indicates  a  rather  severe  mismatch  between 
resources  available  to  the  388th  FW  and  the  day-to-day  missions  it 
is  tasked  to  accomplish — namely,  the  requirement  to  rejuvenate 
human  capital.  The  UTE  [utilization]  rates  are  not  high  enough  to 
maintain  a  healthy  pilot  inventory.  At  the  same  time  as  UTE  rates 
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have  come  down,  TNMCM  [total  not  mission  capable  due  to 
maintenance]  and  TNMCS  [total  not  mission  capable  due  to  supply] 
rates  have  skyrocketed.  Maintenance  manning  is  becoming  less 
experienced  as  junior  personnel  are  pushed  into  the  wing  to 
replace  a  declining  force  of  5-  and  7-levels.  Although  declining  in 
number,  experienced  maintainers  are  spending  more  time 
producing  sorties,  overwhelming  their  ability  to  properly  teach  the  3- 
levels  and  to  upgrade  themselves,  thereby  threatening  the  long¬ 
term  health  of  the  maintainer  inventory.  [Dahlman  and  Thaler, 

2000,  p.  31] 

Although  it  seems  possible  that  Naval  Aviation  organizations  would 
experience  analogous  concerns  when  faced  with  similar  manning  trends,  a 
search  of  literature  did  not  produce  objective  conclusions  to  this  effect. 

3.  NATEC/NAESU  Utilization  Studies 

In  NAESU  Management  of  Technical  Services,  Boynton,  Seiden  and 
Vaughan  [1995]  discuss  Naval  aviation  engineering  and  technical  services  (ETS) 
from  a  management  perspective:  what  it  is  that  tech  reps  do,  who  needs  their 
services,  and  to  what  level  their  customers  are  satisfied  with  these  services.  The 
authors  begin  the  research  by  categorizing  the  various  types  of  engineering  and 
technical  services  from  the  perspectives  of  the  tech  reps  and  their  customers, 
and  find  that  there  is  a  strong  correlation  between  those  activities  deemed 
important  by  both  the  service  providers  and  customers.  The  authors  also  address 
the  significance  of  finding  measures  of  performance  for  ETS,  referencing  the 
occasional  success  in  identifying  specific  cost  avoidance  through  the  use  of  tech 
reps  and  the  high  perceived  value  of  tech  reps,  yet  acknowledging  that  no  single 
measure  has  proved  especially  useful. 

Malcolm  [1995]  explores  NAESU  technical  reports  (NTRs)  and  aircraft 
reliability  and  maintainability  data  to  evaluate  NAESU  performance.  NTRs 
document  information  intended  to  improve  methods  and  eliminate  deficiencies. 
Malcolm  defines  cost  savings  as  a  performance  measure,  and  proposes  that 
NTRs  can  be  studied  to  derive  the  cost  savings  associated  with  the 
implementation  of  the  tech  rep  recommendations.  After  analyzing  several 
hundred  NTRs  over  a  twenty  year  period,  he  concludes  that  an  individual  NTR 
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can  be  used  to  derive  these  cost  savings.  Malcolm  acknowledges  the 
difficulty  in  isolating  the  affects  of  NTRs,  noting  that  sources  of  information  other 
than  NTRs  also  impact  the  change  process. 

Boynton  et  al.  [1998]  continue  previous  studies  and  further  develop 
organizational  measures  of  effectiveness  for  NAESU.  They  conclude  that 
accepted  measures  of  industrial  productivity  are  not  appropriate  measures  of 
effectiveness  of  tech  rep  activities,  primarily  because  tech  rep  “output”  is  not 
defined  in  quantifiable  terms: 

...it  would  seem  that  measures  of  economy  and  measures  of 
effectiveness  could  be  developed  for  NAESU.  Measures  of 
productivity,  however,  would  be  very  difficult  and  arbitrary. 
Production  units  of  input  and  output,  required  to  develop 
input/output  ratios,  are  not  definable  in  this  kind  of  intangible, 
knowledge-intensive  environment.  [Boynton,  1998,  p.  20] 

Instead,  the  authors  focus  on  client  satisfaction,  economic  impact,  and 
contribution  to  NAESU  organizational  objectives.  The  authors  note  that  cost 
savings  realized  by  tech  rep  activity  is  enjoyed  not  by  NAESU/NATEC  but 
instead  by  the  client  customer.  They  also  note  the  difficulty  in  isolating  the  tech 
reps’  contributions  from  other  factors  that  affect  readiness.  The  authors  dismiss 
what  may  seem  to  be  the  obvious  means  of  measuring  tech  rep  effectiveness — a 
controlled  experiment  in  which  such  services  are  available  to  some  squadrons 
and  not  others — due  to  the  predictable  objections  raised  by  units  that  would  be 
adversely  affected  by  such  an  experiment.  Instead,  they  recommend  the 
development  of  a  periodically  administered  random  survey  that  addresses  broad 
categories  of  training,  liaison,  advice,  and  maintenance,  across  all  customer 
categories,  to  identify  indicators  of  service  quality. 

4.  Measuring  the  Value  of  Human  Capital 

Senior  maintenance  personnel  and  tech  reps  are  noteworthy  in  that  they 
have  years  of  experience  to  draw  upon  when  training  less  experienced 
maintainers  and  when  troubleshooting  aircraft  problems.  Much  of  their  value, 
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therefore,  lies  in  what  industry  refers  to  as  “intellectual  capital”,  an  important  but 
difficult  asset  to  quantify.  Wagner  [1998]  leverages  industry  techniques  to  assess 
the  purported  loss  of  intellectual  capital  attributed  to  the  drawdown  of  U.S.  Air 
Force  line  officers  from  1989  to  1997.  He  proposes  to  first  measure  human 
capital  and  then  match  the  Air  Force’s  intellectual  needs  with  its  strategic  plan. 
Wagner  uses  measure  of  human  intellectual  capital  that  are  analogous  to  those 
used  in  industry:  education,  experience,  stability,  growth,  and  efficiency.  He 
concludes  that  increased  trends  in  these  measures  suggest  that  although  overall 
numbers  of  Air  Force  line  officers  decreased  between  1989  and  1997,  the  Air 
Force  managed  their  intellectual  capital  effectively. 

C.  FOCUS  OF  THE  THESIS 

The  primary  objective  of  this  thesis  is  to  identify  squadron  characteristics 
that  are  important  predictors  of  maintenance  performance.  In  other  words,  we 
want  to  determine  which  characteristics  differentiate  the  squadrons  with  respect 
to  a  given  performance  parameter.  We  hope  that  the  process  will  serve  as  proof 
of  concept  and  can  be  used  as  a  tool  to  include  other  squadron  characteristics  as 
data  becomes  available.  A  secondary  objective  is  to  draw  insights  regarding  the 
utility  of  data  currently  being  collected  by  NATEC  and  to  make  recommendations 
for  improvement  if  appropriate.  Specifically,  we  want  to  answer  the  following 
research  questions: 

1.  Which  squadron  characteristics  have  a  detectable  contribution  to  the 
variability  of  the  performance  measure  man-hours  per  maintenance  action ? 

2.  How  much  additional  variability  is  explained  by  the  squadron  that  is  not 
accounted  for  by  the  squadron  characteristics  already  considered? 

3.  Is  there  a  time-of-year  effect  for  the  performance  of  the  squadrons? 

4.  What  additional  metrics  not  currently  available  would  most  likely  be 
useful  in  an  explanatory  model  of  maintenance  performance? 

5.  What  data  collection  methods,  if  any,  would  be  likely  to  improve  the 
ability  of  NATEC  managers  to  correlate  squadron  characteristics  to  tech  rep 
measures  of  performance? 

We  address  the  first  three  questions  in  Chapter  III,  and  the  last  two 
questions  in  Chapter  IV. 
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D.  SCOPE  OF  RESEARCH 

We  address  our  research  objectives  as  they  pertain  to  Marine  Corps 
squadrons  that  operate  the  F/A-18,  a  multi-role  strike  fighter  flown  in  both  the 
Navy  and  Marine  Corps.  Since  the  Navy  and  Marine  Corps  differ  somewhat  in 
their  missions,  operational  cycles,  and  other  factors,  we  control  for  the  “service 
effect”  by  focusing  on  Marine  Corps  squadrons.  Similarly,  differences  found  in 
reserve  and  training  squadrons  are  controlled  by  considering  only  active  duty 
fleet  squadrons.  The  current  Marine  Corps  inventory  consists  of  thirteen  active 
duty  F/A-18  squadrons  of  aircraft  model  types  A,  C,  and  D  that  are  included  in 
this  analysis. 


Figure  2  Marine  Corps  F/A-18  Fighter  Attack  Aircraft. 

F/A-18  aircraft  such  as  that  pictured  here  are  operated  by  squadrons  stationed  at 
Marine  Corps  Air  Station  (MCAS)  Miramar,  CA  and  MCAS  Beaufort,  SC.  VMFA- 
212  is  permanently  assigned  to  MCAS  Iwakuni,  Japan. 

Although  tech  reps  interact  with  maintenance  personnel  at  both  the 
intermediate  level  and  organizational  (squadron)  level,  we  limit  our  analysis  to 
organizational  level  (O-level)  maintenance.  Finally,  since  NATEC  has 
documented  the  activity  of  its  tech  reps  in  ELAR  since  August  2003,  analysis  that 
includes  tech  rep  variables  is  limited  to  the  time  window  August  2003  to  May 
2005.  Data  pertaining  to  variables  other  than  tech  rep  activities  encompasses,  at 
a  minimum,  this  time  period. 
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E.  ORGANIZATION 

Chapter  II  describes  the  objective  and  methodology  used  in  the  collection 
of  data  to  support  the  analysis.  After  defining  the  metrics  that  describe  the 
squadrons’  characteristics,  we  identify  the  sources  of  the  data  that  allow  us  to 
derive  these  metrics.  Chapter  III  begins  with  an  exploration  of  the  data  as  time 
series  plots.  After  eliminating  redundant  variables  we  employ  regression  analysis 
techniques  to  identify  those  characteristics  that  are  related  to  performance. 
Chapter  IV  summarizes  the  work  and  makes  recommendations  for  further  study. 
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II.  DATA  COLLECTION 


A.  OBJECTIVES 

The  purpose  of  this  thesis  is  to  identify  squadron  characteristics  that  can 
explain  the  performance  of  its  maintenance  crew.  The  data  collection  effort 
supported  our  analysis  by  focusing  on  data  elements  that  describe  the  squadron 
characteristics  in  quantifiable  terms,  and  which  describe  the  performance  output 
of  a  squadron’s  maintenance  department. 

Our  goal  is  to  express  the  various  characteristics  of  the  squadrons  in 
terms  of  personnel  makeup,  aircraft  inventory,  maintenance  activity,  and 
operational  activity.  In  this  respect  we  are  describing  what  the  squadrons  are 
doing,  who  they’re  doing  it  with,  and  what  they’re  doing  it  with.  We  want  to 
quantify  performance  in  terms  of  mission  capability  rates,  times  to  repair,  and  the 
frequency  of  certain  types  of  repair.  Low  capability  rates  may  signify  that  the 
squadron  is  not  getting  sufficient  flight  hours  from  the  aircraft  in  its  possession. 
Long  times  to  repair  may  indicate  poor  maintenance  management,  slow  aircraft 
turnaround  activities,  or  even  a  lack  of  personnel  capabilities.  High  frequency  of 
repair  may  be  more  indicative  of  a  reliability  problem  than  of  a  maintenance 
problem,  so  we  limited  our  scope  to  those  types  of  repair  that  are  indicative  of 
maintenance  performance.  Finally,  we  need  to  consider  these  factors  in  the 
context  of  changing  operational  tempo  and  deployments,  so  we  obtained  data 
that  describe  squadrons  in  operational  terms  as  well. 

B.  METHODOLOGY 

We  began  the  data  collection  effort  by  identifying  metrics  that  capture  the 
squadron  characteristics  that  we  want  to  quantify.  With  these  metrics  in  mind,  we 
identified  data  sources  that  contain  these  metrics  (or  the  raw  data  that  allow  them 
to  be  derived)  for  the  squadrons  under  consideration  in  this  study.  From  each  of 
these  sources,  we  collected  time  series  data  encompassing,  at  a  minimum,  the 
period  August  2003  through  May  2005.  Since  each  of  the  data  sources  provides 
data  at  differing  levels  of  detail,  we  decided  on  a  time  unit  that  facilitates  a  useful, 
common  level  of  aggregation.  Many  available  metrics  were  already  aggregated 
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on  a  monthly  basis,  so  further  decomposition  to  a  weekly  or  daily  interval  was 
avoided.  The  data  also  needed  to  be  aggregated  to  the  appropriate 
organizational  level.  Although  some  detailed  metrics  attribute  maintenance 
activity  to  a  single  aircraft,  the  squadron  is  the  smallest  maintenance  organization 
to  which  all  our  metrics  apply.  The  data  were  aggregated  to  the  squadron  level 


and  filtered  to  include  only  the  following 

Third  Marine  Aircraft  Wing 
Marine  Aircraft  Group  11 
Marine  Corps  Air  Station  Miramar,  CA 

VM  FA-232 

VM  FA-3 14 

VM  FA-323 

VMFA(AW)-121 

VMFA(AW)-225 

VMFA(AW)-242 


squadrons: 

Second  Marine  Aircraft  Wing 
Marine  Aircraft  Group  31 
Marine  Corps  Air  Station  Beaufort,  SC 

VMFA-1 15 

VMFA-122 

VM  FA-251 

VM  FA-3 12 

VMFA(AW)-224 

VMFA(AW)-332 

VMFA(AW)-533 


Table  1.  CONUS-Based  Active  Duty  Marine  F/A-18  Squadrons 


For  data  elements  that  are  measured  over  a  large  number  of  individuals  or 
aircraft  in  a  given  month,  we  use  summary  statistics  such  as  quartiles  to 
condense  the  data  into  single  values  per  squadron  per  month.  Each  metric, 
therefore,  takes  the  form  where  s  represents  the  squadron  and  /represents 

the  month.  For  certain  metrics  we  are  constrained  by  the  months  for  which  data 
was  not  collected  or  made  available  to  us.  Since  we  are  analyzing  a  collection  of 
factors  across  a  consistent  timeframe,  we  limited  our  data  collection  effort  to  a 
time  frame  that  was  common  to  each. 

C.  METRICS  AND  THEIR  SOURCES 

We  identify  sources  of  data  that  allow  us  to  develop  a  set  of  metrics  that 
quantify  the  characteristics  of  the  squadrons  in  terms  of  operational  activity, 
maintenance  activity,  personnel  makeup,  and  aircraft  inventory.  These  data 
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sources  are  not  linked  and  are  not  designed  to  share  data  or  commonality  of  data 
format.  As  such,  we  develop  metrics  that  can  be  brought  together  on  a  common 
scale  or  time  increment.  Each  metric  is  categorized  by  its  relation  to  operations, 
maintenance,  personnel  composition,  or  aircraft  inventory. 

1.  Flight  Operations  and  Maintenance  Metrics 

Flight  operations  metrics  are  characterized  by  their  relationship  to  the  daily 
flight  schedule.  We  expect  the  volume  of  maintenance  activity  to  vary  with  the 
demands  of  flight  operations.  Although  operational  demands  may  be  beyond  the 
control  of  the  maintenance  department,  we  use  operational  metrics  to  normalize 
maintenance  statistics  to  a  common  scale. 

We  accessed  the  Navy’s  Maintenance  Data  System  (MDS)  to  collect  the 
data  elements  used  to  describe  maintenance  and  performance  and  flight 
operations.  MDS,  managed  by  Naval  Air  Systems  Command  (NAVAIR),  is  a 
management  information  system  designed  to  provide  statistical  data  on 
equipment  maintainability  and  reliability,  configuration,  mission  capability  and 
utilization,  material  usage  and  non-availability,  and  maintenance  and  material 
processing  times  and  costing.  Maintainers  and  aircrew  input  data  into  MDS  using 
the  Naval  Aviation  Logistics  Command  Information  System  (NALCOMIS),  the 
primary  maintenance  interface  that  integrates  all  maintenance  functions  and 
allows  managers  to  visualize  critical  maintenance  trends  through  pre-designed  or 
customizable  reports.  For  access  to  data  maintained  in  NALCOMIS,  we  used  the 
web-based  Decision  Knowledge  Programming  for  Logistics  Analysis  and 
Technical  Evaluation  (DECKPLATE),  which  provides  report  and  query 
capabilities  of  Naval  Aviation  logistics  and  flight  event  data  to  compile  the 
maintenance  and  operational  characteristics  of  each  squadron.  For  each  of  the 
metrics  described  previously,  we  collected  data  for  the  period  October  2002  to 
April  2005. 

From  these  data  sources,  we  identify  operational  and  maintenance 
metrics  that  allow  us  to  quantify  performance  output  in  measurable  terms.  We 
group  the  operational  metrics  as  measures  of  utilization  and  operational  tempo, 

and  group  the  maintenance  metrics  as  measures  of  maintainability  and  reliability. 
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a. 


Measures  of  Utilization 


Each  squadron  operates  its  aircraft  at  rates  required  for  training  or 
for  contingency  operations.  We  anticipated  the  need  to  control  for  these  factors 
and  therefore  collected  measures  of  utilization:  flights,  flight  hours,  utilization,  and 
deployment  status. 

Flights  and  Flight  Hours.  These  metrics  are,  respectively,  the 
total  number  of  flights  and  flight  hours  flown  by  the  entire  squadron  during  the 
designated  time  period.  The  flights  metric  is  reported  as  an  integer  value,  and  the 
flight  hours  metric  is  reported  to  the  nearest  tenth  of  an  hour.  We  categorize  both 
flights  and  flight  hours  as  performance  metrics. 

Utilization.  The  utilization  metric  is  the  average  number  of  hours 
flown  per  aircraft  during  a  given  month,  and  is  reported  to  the  nearest  tenth  of  an 
hour.  We  categorize  this  metric  as  a  performance  metric. 

b.  Measures  of  Operational  Tempo 

Arguably,  squadrons  that  are  deployed  or  approaching  their 
deployment  date  experience  higher  supply  prioritization,  better  support 
equipment,  personnel  augmentation,  boosted  morale,  and  a  higher  sense  of 
urgency.  The  metric  deployment  status  attempts  to  capture  these  factors.  This 
categorical  metric  distinguishes  between  the  following  deployment  modalities: 

•  Continental  U.S.  (CONUS) 

•  Unit  Deployment  Program  (UDP) 

•  U.S.  Navy  aircraft  carrier  (CVN) 

•  Combat  contingencies,  which  during  the  time  frame  of  this  thesis 
consisted  either  of  Operation  Enduring  Freedom  or  Operation  Iraqi 
Freedom. 

To  collect  the  data  that  would  allow  us  to  categorize  the 
deployment  status  of  the  squadrons,  we  turned  to  various  unclassified,  publicly 
accessible  sources  of  squadron  historical  records  such  as  press  releases 
[Pasnik,  2005],  official  squadron  web  sites  and  widely-used  military  synopsis  web 
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sites  [GlobalSecurity.org,  2005],  Use  of  deployment  status  in  our  analyses  allows 
us  to  control  for  the  operational  tempo  of  a  squadron  when  we  compare  its 
performance  metrics  to  those  of  other  squadrons.  We  categorize  deployment 
status  as  an  operational  metric. 

c.  Measures  of  Availability 

An  important  descriptor  of  maintenance  performance  is  the 
proportion  of  aircraft  not  available  for  operations  or  training.  This  measure  can  be 
delineated  by  the  specific  reason  the  aircraft  is  not  available  for  training  - 
ongoing  corrective  or  preventative  maintenance,  or  supply  delays. 

Not  Mission  Capable-Maintenance  (NMCM)  rate.  NMCM  is  the 
proportion  of  total  reported  time  that  a  squadron’s  aircraft  are  not  mission 
capable  due  to  maintenance  actions  required.  Low  NMCM  rates  are  desirable, 
whereas  high  NMCM  rates  may  indicate  poor  maintenance  management, 
capabilities,  prioritization,  or  flight  operations  coordination  practices.  NMCM  rates 
can  be  further  delineated  by  non-mission  capability  resulting  from  scheduled 
maintenance  ( NMCMS )  and  unscheduled  maintenance  ( NMCMU ).  Scheduled 
maintenance  can  be  planned  and  managed  by  effective  maintenance  managers. 
Unscheduled  maintenance  results  from  unanticipated  breaks.  We  categorize  this 
family  of  mission-capability  measures  as  performance  metrics. 

Not  Mission  Capable-Supply  (NMCS)  rate.  NMCS  is  the 
proportion  of  total  possessed  time  that  the  squadron’s  aircraft  are  not  mission 
capable  due  to  supply  reasons.  Although  some  of  this  time  is  attributable  to 
supply  shortages,  this  metric  can  also  be  influenced  by  maintenance  practices. 
Maintenance  analysts  can  identify  which  parts  are  responsible  for  putting  an 
aircraft  in  an  NMCS  status.  Some  parts  are  well  known  for  being  on  long  back¬ 
orders,  while  others,  although  usually  available,  are  frequently  ordered.  We 
categorize  NMCS  as  a  performance  metric. 

d.  Measures  of  Maintainability 

Man  Hours  per  Flight  Hour.  This  metric  provides  an  indication  of 
the  amount  of  maintenance  effort  associated  with  each  flight  hour  flown  by  the 

squadron.  A  large  value  of  Man  Hours  per  Flight  Hour  may  be  associated  with 
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older,  less  reliable  aircraft  that  require  additional  maintenance,  or  perhaps  an 
indication  of  a  less  capable  manpower  base.  We  categorize  this  metric  as  a 
performance  metric. 

Man  Hours  per  Maintenance  Action.  This  metric  quantifies  the 
average  number  of  man  hours  required  to  complete  a  maintenance  action,  and  is 
calculated  by  dividing  the  total  number  of  man  hours  by  the  number  of 
maintenance  actions  for  each  of  the  31  months  in  the  data  query.  Since  many 
maintenance  actions  result  from  normal  equipment  failure,  while  others  are 
operationally  driven  (in  that  they  are  triggered  by  accumulated  flight  hours),  the 
number  of  maintenance  actions  is  driven  largely  by  circumstances  beyond  the 
control  of  the  maintenance  department.  The  number  of  maintenance  actions, 
therefore,  is  not  by  itself  a  good  indicator  of  the  health  of  the  maintenance 
department.  Instead,  we  use  this  number  as  a  scaling  factor  to  normalize  other 
measures,  such  as  maintenance  man  hours,  to  a  common  scale,  which  we 
quantify  as  Man  Hours  per  Maintenance  Action.  We  categorize  this  metric  as  a 
performance  metric. 

TD  Hours.  On  occasion,  squadrons  are  instructed  to  incorporate 
changes,  such  as  airframe  modifications  and  avionics  upgrades,  to  the  aircraft  in 
their  possession.  These  changes  are  known  as  technical  directives  (TDs).  TD 
hours  quantifies  the  amount  of  time  dedicated  by  a  squadron  to  address  the 
changes  required  by  TDs.  We  separate  out  this  type  of  maintenance  activity 
because  it  may  require  additional  expertise  or  perhaps  external  support.  Perhaps 
squadrons  with  higher  overall  capability  levels  will  accomplish  TDs  in  shorter 
amounts  of  time.  TD  hours  is  quantified  in  units  of  hours  for  a  specified  squadron 
and  month.  We  categorize  TD  hours  as  a  performance  metric. 

To  collect  the  TD  hours  metric  we  accessed  the  Technical  Directive 
Status  Accounting  (TDSA)  database  and  collected  25  months  of  data  from  April 
2003  to  April  2005.  This  metric  is  reported  in  hours  to  the  nearest  tenth. 
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e.  Measures  of  Reliability 

Metrics  that  quantify  the  frequency  of  repair  are  often  used  to 
evaluate  system  reliability.  Instead,  we  limited  our  use  of  repair-frequency 
metrics  to  those  that  characterize  maintenance  performance  rather  than  aircraft 
reliability. 


Carmibalizations  per  100  hrs.  This  metric  is  the  average  number 
of  cannibalizations  per  100  hours.  A  cannibalization  is  the  removal  of  a 
serviceable  part  from  an  aircraft  to  replace  an  unserviceable  part  of  another 
aircraft.  Curtin  [2001]  testifies  to  Congress  the  effects  of  cannibalizations: 

Cannibalizations  have  several  adverse  impacts.  They  increase 
maintenance  costs  by  increasing  workloads,  may  affect  morale  and 
the  retention  of  personnel,  and  sometimes  result  in  the 
unavailability  of  expensive  aircraft  for  long  periods  of  time. 
Cannibalizations  can  also  create  unnecessary  mechanical 
problems  for  maintenance  personnel.  [Curtin,  2001,  p.  2], 

Since  cannibalizations  occur  when  the  required  parts  are  not 
immediately  available  from  the  supply  system,  this  metric  is  often  used  as  a 
measure  of  supply  effectiveness.  A  rising  value  usually  results  in  an  increase  in 
the  number  of  man-hours  required  to  achieve  the  same  utilization  rates  and  flight 
hours  and  risks  causing  additional  problems  to  the  cannibalized  aircraft.  We 
categorize  this  metric  as  a  performance  metric. 

A799s  per  Flight  Hour.  This  measure  quantifies  the  frequency  of 
circumstances  where  maintenance  personnel  are  unable  to  diagnose  a  problem 
that  was  identified  by  the  aircrew.  Such  cases  are  documented  in  NALCOMIS 
with  the  Maintenance  Action  Code  A799,  which  signifies  that  the  maintenance 
personnel  could  find  no  defect  and  took  no  corrective  action.  In  some  cases,  this 
represents  a  failure  on  the  part  of  maintainers  to  resolve  a  problem.  In  other 
cases,  it  represents  a  failure  on  the  part  of  the  aircrew  to  appropriately  describe 
the  problem,  or  simply  the  reality  that  environmental  conditions  in  the  repair 
facility  are  not  the  same  as  those  under  which  the  failure  occurred.  We 
categorize  this  metric  as  a  performance  metric. 
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2.  Personnel  Metrics 

The  maintenance  organization  of  a  squadron  is  staffed  to  handle  most  of 
the  squadron’s  maintenance  requirements.  The  ability  of  a  squadron  to  meet 
operational  demands  lies  largely  with  the  capabilities  of  the  maintenance 
personnel.  We  attempt  to  quantify  their  capability  with  a  variety  of  personnel 
metrics  that  may  characterize  capability  to  some  degree.  We  develop  personnel 
metrics  with  enough  variability  to  help  explain  differences  in  maintenance 
performance  on  a  monthly  basis.  Manpower  metrics  that  may  prove  useful  in 
quantifying  a  unit’s  capabilities  are  those  that  capture  the  maintainers’  collective 
experience  level,  as  measured  by  their  years  or  months  of  service  or  their  time  in 
the  current  squadron.  These  variables,  while  not  direct  measurements  of 
individual  capability,  may  serve  as  useful  representatives  for  such  a  measure.  In 
addition  to  these  measures  of  experience,  we  express  a  unit’s  capability  with 
measures  of  stability. 

The  personnel  metrics  we  use  in  the  analysis  were  derived  from  records 
extracted  from  the  Marine  Corps  Total  Force  System  (MCTFS).  MCTFS  is  the 
single,  integrated,  personnel  and  pay  system  supporting  both  Active  and  Reserve 
components  of  the  Marine  Corps.  To  derive  the  desired  metrics,  we  created 
monthly  squadron  personnel  snapshots  for  a  24-month  period  beginning  May 
2003.  We  compiled  the  records  of  those  personnel  that  were  members  of  the 
designated  squadrons  during  each  of  those  months.  In  addition  to  each 
individual’s  name  and  unit,  we  collected  the  following  data  from  MCTFS  to  allow 
for  calculation  of  desired  metrics: 

•  Military  Occupational  Specialty  (MOS). 

•  Date  arrived  at  current  duty  station  (ArrPermDutySta) 

•  Date  departed  previous  duty  station  (DepPermDutySta) 

•  Rank  (grade). 

•  Months  served  on  active  duty  (Active  Service). 

a.  Measures  of  Experience 

We  express  the  capability  of  a  squadron,  to  a  certain  extent,  in 
terms  of  the  collective  experience  of  the  maintenance  personnel. 
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Months  in  Service  (lower  quartile,  median,  upper  quartile). 

These  three  metrics  quantify  the  distribution  of  experience  of  the  maintainers, 
using  time  in  service  as  a  measure  of  experience.  The  lower  quartile,  or  25th 
percentile,  is  the  value  such  that  approximately  one-quarter  of  the  sample  lies 
below  it.  The  upper  quartile,  or  75th  percentile,  is  the  value  such  that 
approximately  one-quarter  of  the  sample  lies  above  it.  About  half  of  the  sample, 
therefore,  lies  between  the  lower  and  upper  quartiles.  The  median,  or  50th 
percentile,  is  the  value  such  that  about  half  of  the  sample  lies  below  and  half  of 
the  sample  lies  above  that  value.  A  Marine  Corps  F/A-18  squadron  is  assigned 
approximately  150  maintainers,  but  the  actual  number  of  maintainers  in  a  given 
squadron  fluctuates  somewhat,  especially  during  the  months  preceding  or 
immediately  following  a  deployment.  We  aggregate  the  monthly  data  of  the 
individual  maintainers  into  quartiles  to  preserve  some  information  about  the 
distribution  of  experience  in  the  squadron,  resulting  in  three  metrics  for  each 
squadron  per  month.  The  unit  of  measure  for  this  metric  is  months.  We 
categorize  this  metric  as  a  descriptive  metric. 

Months  in  Squadron  (lower  quartile,  median,  upper  quartile). 

These  three  metrics  quantify  the  distribution  of  experience  of  the  maintainers  in 
terms  of  their  time  in  their  current  squadron.  We  recognize  that  individuals’  time 
in  the  service  may  be  interrupted  by  duties  unrelated  to  maintenance  work.  This 
metric  eliminates  periods  of  interruption  by  considering  only  the  time  spent  in  a 
particular  squadron.  As  with  the  months  in  service  metrics,  we  aggregate  the 
monthly  data  of  the  individual  maintainers  into  quartiles,  resulting  in  three 
separate  metrics.  The  unit  of  measure  for  this  metric  is  months.  We  categorize 
this  metric  as  a  descriptive  metric. 

b.  Measures  of  Stability 

The  personnel  that  comprise  the  squadrons’  maintenance 
organizations  change  on  a  daily  basis.  We  characterize  manning  stability  by  the 
frequency,  magnitude,  and  trends  of  these  personnel  changes. 
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Turnover  rate.  This  metric  expresses  stability  in  terms  of  the 
number  of  individuals  entering  and  leaving  the  organization  as  a  proportion  of  the 
total  number  of  individuals  in  the  organization.  We  categorize  this  metric  as  a 
descriptive  metric. 

3.  Aircraft  Inventory  Metrics 

Although  each  squadron  is  assigned  twelve  F/A-18  aircraft,  the 
type/model/series  of  these  aircraft  differ  between  squadrons,  as  do  their  accrued 
use  and  age.  To  develop  the  metrics  needed  to  describe  inventory 
characteristics,  we  obtained  monthly  reports  from  NAVAIR  which  were 
compilations  of  data  extracted  from  the  Aircraft  Inventory  Readiness  and 
Reporting  System  (AIRRS)  and  SAFE  [Kaitchuk,  R.,  personal  email 
correspondence,  July  10,  2005],  AIRRS  provides  on-line  access  to  aircraft 
inventory,  readiness,  and  flight  utilization  data.  SAFE  is  the  structural  appraisal  of 
fatigue  effects.  Output  from  AIRRS  was  made  available  to  us  in  a  series  of  files, 
each  of  which  was  a  monthly  snapshot  of  inventory  data.  Since  each  squadron 
operates  multiple  aircraft,  we  used  the  mean  to  represent  the  distribution  of  each 
squadron’s  inventory  metrics  for  a  given  month.  The  following  metrics  are  used  to 
quantify  these  inventory  characteristics. 

Type  Equipment  Code  (TEC).  Each  squadron  operates  a  single 
type/model/series  of  aircraft.  Varying  aircraft  types  may  demand  different 
maintenance  efforts,  so  we  categorize  them  appropriately.  TEC  categorizes  the 
aircraft  type  with  a  four-letter  code;  the  F/A-18A,  F/A-18C,  and  F/A-18D  are 
represented  by  the  codes  “AMAA,”  “AMAF,”  and  “AMAG,”  respectively.  We 
categorize  this  metric  as  a  descriptive  metric. 

Airframe  Hours.  This  metric  indicates  the  accrued  flight  hours  that  have 
been  accumulated  over  the  life  of  the  aircraft  during  its  lifetime,  averaged  across 
the  inventory  of  aircraft. 

Airframe  Months  in  Service.  This  field  indicates  the  total  number  of 
months  that  each  aircraft  in  the  squadron  has  been  in  operation  since  entering 
service. 
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4.  Engineering  and  Technical  Support  (ETS)  Metrics 

As  described  in  the  Background  section,  ETS  support  is  provided  by  tech 
reps  that  are  available  to  each  squadron.  NATEC  uses  the  ELAR  database  to 
track  initiated  requests  for  ETS  assistance  through  execution  and  completion. 
ELAR  records  date  back  to  August  2003.  We  collected  all  records  available, 
filtered  to  include  only  those  associated  with  F/A-18  organizations,  and 
developed  metrics  that  allow  us  to  include  the  frequency  of  these  support  assists 
in  our  analysis. 

Records  per  Month.  This  metric  quantifies  the  volume  of  ETS  activity  by 
quantifying  the  number  of  assists  provided  to  the  squadrons.  We  categorize  this 
metric  as  a  descriptive  metric. 

5.  Location  and  Organization. 

Although  we  want  to  identify  underlying  causes  that  result  in  performance 
variation,  we  might  find  that  performance  is  explained  in  part  by  location  and 
factors  associated  with  different  operating  bases,  such  as  the  variety  of  support 
structures  or  local  policies  not  otherwise  identified  in  this  analysis.  Table  1.  lists 
the  squadrons  and  their  associated  home  bases.  Since  only  two  bases  are 
associated  with  the  squadrons  under  investigation,  we  can  use  a  two-level  factor 
to  identify  the  observation  as  either  “Beaufort”  or  “Miramar.” 

We  are  trying  to  explain  variability  in  squadron  performance  by  quantifying 
the  inherent  characteristics  of  the  units  with  metrics  that  are  applicable  to  all 
squadrons.  However,  we  have  to  consider  the  possibility  that  some  of  the 
variance  is  described  by  other  variables  not  yet  considered.  A  categorical 
variable  indicating  the  squadron,  called  organization,  will  be  used  to  capture 
additional  variance  due  to  squadron  differences  that  are  not  captured  by  the 
operational,  personnel  or  inventory  variables  described  in  this  chapter.  We 
should  note  that  when  using  organization  as  an  indicator,  we  identify  location  and 
type  equipment  code  (TEC)  implicitly,  since  organization  uniquely  identifies  its 
location  and  TEC. 


29 


D.  COMPILATION  OF  DATA 

Table  2.  summarizes  the  metrics  and  the  sources  of  data  that  we  used  to  derive 
them. 


Data  Source 

Group 

Metric 

Type 

NALCOMIS 

Measures  of  Utilization 

Flights  and  Flight  Hours 

Performance 

Utilization 

Performance 

Measures  of  Availability 

NMCM 

Performance 

NMCS 

Performance 

Measures  of 

Man-hours  per  flight  hour 

Performance 

Maintainability 

Man-hours  per  maintenance 

action 

Performance 

TD  hours 

Performance 

Measures  of  Reliability 

Cannibalizations  per  flight  hour 

Performance 

A799s  per  flight  hour 

Performance 

MCTFS 

Measures  of  Experience 

Months  in  service  quartiles 

Descriptive 

Months  in  squadron  quartiles 

Descriptive 

Measure  of  Stability 

Turnover  rate 

Descriptive 

AIRRS 

Aircraft  Type 

Type  equipment  code 

Descriptive 

Measures  of  Aircraft  Age 

Airframe  hours 

Descriptive 

Airframe  months  in  service 

Descriptive 

ELAR 

Measure  of  ETS  Activity 

Records  per  month 

Descriptive 

Various 

Measure  of  Ops  Tempo 

Deployment  Status 

Descriptive 

N/A 

Measure  of  Environment 

Location 

Descriptive 

Table  2.  Groups,  Types  and  Sources  of  Metrics 
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The  number  of  observations  n  is  determined  by  the  number  of  months  (31) 
multiplied  by  the  number  of  squadrons  under  investigation  (13)  for  a  total  of  n  = 
403  observations  of  the  form  Xist ,  where  i  =  variable  number,  s  =  squadron, 

and  t-  month.  For  ease  of  manipulation  we  arranged  the  observations  in  a  table 
of  n  observations  (rows)  of  24  variables  (columns).  Some  of  the  observations 
contain  missing  values  for  certain  variables.  Only  209  observations  have  no 
missing  values.  A  portion  of  the  table  is  shown  in  Appendix  C. 


31 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


32 


III.  ANALYSIS 


A.  OBJECTIVE 

The  objective  of  the  analysis  phase  is  to  describe,  in  mathematical  terms, 
the  relationship  between  active  component  Marine  Corps  F/A-18  squadrons’ 
descriptive  metrics  and  their  performance  metrics,  which  we  define  in  Chapter  II. 

B.  APPROACH 

We  begin  by  analyzing  trends  and  variability  of  the  performance  and 
descriptive  metrics  with  the  use  of  time  series  plots  and  histograms.  An 
understanding  of  trends  and  patterns  provides  insights  to  where  measurable 
differences  between  squadrons  may  exist,  and  where  there  is  correlation 
between  factors.  After  discussing  the  performance  metrics,  we  focus  on  a  single 
performance  metric,  man-hours  per  maintenance  action,  for  further  analysis. 
Through  the  use  of  correlation  analysis,  we  limit  the  complexity  of  the  model¬ 
building  problem  by  reducing  the  set  of  potential  predictor  variables  to  a  smaller 
representative  subset.  We  then  direct  our  analysis  to  answer  the  following 
research  questions: 

1.  Which  squadron  characteristics  have  a  detectable  contribution  to 
the  variability  of  the  performance  measure  man-hours  per  maintenance  action ? 

2.  How  much  additional  variability  is  explained  by  the  squadron  that  is 
not  accounted  for  by  the  squadron  characteristics  already  considered? 

3.  Is  there  a  delayed  response  (lag)  between  any  of  the  descriptive 
(predictor)  variables  and  the  performance  measure  (response)  variable? 

4.  Is  there  a  time-of-year  effect  for  the  performance  of  the  squadrons? 

To  answer  the  first  question,  we  use  predictor  variables  and  a  response 
variable  in  regression  analysis  to  construct  a  linear  combination  of  descriptive 
variables  that  best  explains  the  variability  of  the  squadrons’  performance.  To 
answer  the  second  question,  we  add  the  categorical  variable  organization  to  the 
resulting  model,  to  determine  whether  the  predictive  power  of  the  model  is 
increased;  and  if  so,  to  what  extent. 
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C.  TIME  SERIES  EXPLORATION 
1.  Performance  Measures 

We  begin  by  identifying  those  metrics  that  best  describe  maintenance 
performance.  Although  we  have  access  to  numerous  metrics  that  measure 
maintenance  activity,  many  of  them  simply  count  system  failures  or  maintenance 
actions  and  their  frequency.  As  such,  they  are  measures  of  reliability  rather  than 
of  maintenance  performance,  designed  to  identify  to  supply-chain  analysts  those 
critical  components  that  may  be  exhibiting  high  rates  of  failure.  These  higher 
failure  rates  could  be  a  result  of  factors  that  do  not  reflect  maintainer  capability. 
Instead,  we  focus  on  those  metrics  that  quantify  capabilities  of  the  personnel 
performing  the  repairs  once  the  failures  have  occurred.  We  will  limit  our  initial 
analysis  to  the  following  performance  metrics  as  response  variables:  not  mission 
capable  due  to  maintenance,  unscheduled  (NMCMU),  man  hours  per  flight  hour, 
man  hours  per  maintenance  action,  A799  actions  per  flight  hour,  and 
cannibalization  actions  per  flight  hour.  After  describing  the  distributions  and 
variability  of  these  five  response  variables,  we  will  use  Man  Hours  per 
Maintenance  Action  for  more  rigorous  analysis  -  as  a  case  study  and  proof  of 
concept  for  addressing  the  objectives  and  primary  research  questions  posed  in 
Chapter  I. 

a.  Not  Mission  Capable  Due  to  Maintenance,  Unscheduled 
(NMCMU) 

NMCM  is  the  proportion  of  total  Equipment  in  Service  (EIS)  time 
that  an  aircraft  is  not  fully  mission-capable  due  to  ongoing  maintenance  activity. 
NMCMU  is  more  narrowly  defined  by  maintenance  of  the  unscheduled  variety. 
Figure  3  depicts  the  monthly  movement  of  this  metric  for  the  thirteen  active  duty 
Marine  Corps  F/A-18  squadrons.  NMCMU  ranges  from  0  to  0.3,  with  an  average 
value  of  0.12,  across  the  squadrons  and  timeframe  considered. 
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Figure  3  NMCMU  Time  Series  for  Active  Duty  Marine  Corps  F/A-18 

Squadrons. 

Each  panel  represents  31  months  of  data  for  each  squadron.  Squadron  labels 
appear  above  each  time  series  plot.  NMCMU  is  expressed  as  a  proportion  on  a 
scale  from  0  to  1. 

Figure  3  indicates  that  there  is  variability  in  NMCMU  both  within  squadrons  and 
between  squadrons.  In  some  units,  such  as  VMFA121  and  VMFA251,  we  see 
upward  movement  of  NMCMU  for  the  months  between  October  2002  and  April 
2005,  which  is  not  a  desirable  trend.  VMFA242,  on  the  other  hand,  exhibits  a 
decreasing  trend.  The  graph  does  not  indicate  obvious  dependence  between 
squadrons,  but  more  rigorous  tests  for  correlation  will  be  conducted  later  in  this 
chapter. 
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b.  Man  Hours  per  Flight  Hour  and  Man  Hours  per 
Maintenance  Action 

The  man  hours  per  flight  hour  metric  is  calculated  by  dividing  the 
total  maintenance  man  hours  in  a  given  month  by  the  flight  hours  flown  in  that 
same  period.  We  expect  a  squadron  that  flies  more  flight  hours  to  experience 
higher  demands  for  both  corrective  and  preventative  maintenance.  By 
normalizing  the  man-hour  data  with  the  number  of  flight  hours,  we  control  for  the 
tendency  of  failures,  maintenance  actions,  and  therefore  man-hours,  to  increase 
with  flight  hours.  In  this  way  we  hope  to  isolate  personnel  capability  from 
reliability  factors.  A  problem  with  the  use  of  flight  hours  as  a  scaling  factor  arises 
when  extremely  low  values  of  flight  hours  destabilize  those  metrics  with  flight 
hours  in  the  denominator.  If  we  use  maintenance  actions  rather  than  flight  hours 
to  normalize  the  man  hours  data,  we  quantify  the  average  number  of  man  hours 
it  takes  the  squadron  to  complete  each  maintenance  action  and  isolate 
maintenance  activity  from  an  operational  factor.  An  added  benefit  of  scaling  with 
maintenance  actions  is  that  we  do  not  see  extremely  low  values  in  the 
denominator  of  the  man  hours  per  maintenance  action  ratio.  Figure  4  depicts  the 
movement  of  man-hours  per  flight  hour  and  man-hours  per  maintenance  action 
on  the  same  time  series  plot. 
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Figure  4  Man-Hours  per  Flight  Hour  and  Man  Hours  per  Maintenance  Action 

Time  Series  by  Squadron. 

Each  panel  shows  31  months  of  data  for  man-hours  per  flight  hour  and  man-hours  per 
maintenance  action.  Those  months  with  fewer  than  50  flight  hours  have  been  omitted, 
since  they  tend  to  distort  the  effects  of  data  normalized  by  flight  hours. 

Similar  movement  between  the  time  series  plots  of  Figure  4 
indicates  that  man-hours  per  maintenance  action  and  man-hours  per  plight  hour 
may  be  correlated  metrics,  and  our  choice  of  which  is  more  suitable  as  a 
performance  measure  may  depend  on  a  more  nuanced  understanding  of  how 
they  are  derived.  Values  of  man-hours  per  flight  hour  range  from  0  to  34.8,  with  a 
mean  of  7.02.  Lower  values  of  this  metric  suggest  a  more  efficient  work  force, 
perhaps  a  result  of  more  experienced  or  better  trained  personnel.  Some  of  the 
squadrons,  such  as  VMFA332  and  VMFA232,  appear  to  exhibit  increasing 
values  in  these  two  performance  metrics,  which  is  an  undesirable  trend.  Other 

squadrons,  such  as  VMFA224,  had  individual  months  during  which  the  man- 
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hours  metrics  were  remarkably  higher  than  average.  In  the  next  section,  we  will 
attempt  to  identify  those  squadron  characteristics  that  explain  the  monthly 
variability  and  long-term  trending  of  the  man-hours  per  maintenance  action 
metric. 

c.  A  799s  per  Flight  Hour 

As  described  in  Chapter  II,  A799s  are  maintenance  actions  in  which 
the  maintainers  could  not  identify  the  problem  and  took  no  further  action.  If  a 
malfunction  reported  by  aircrew  is  not  identified  or  duplicated  by  maintenance 
personnel,  the  aircraft  may  be  determined  to  be  safe  for  flight  and  the  action 
marked  complete  with  an  A799  code.  In  some  cases,  this  represents  a  failure  on 
the  part  of  maintainers  to  resolve  a  problem,  while  at  other  times  the  error  lies 
with  the  aircrew  in  poor  communication  or  misdiagnosis  of  the  problem.  Figure  5 
depicts  the  movement  of  this  metric  over  time  within  squadrons  and  their 
corresponding  least-squares  trend  lines. 
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Figure  5  A799  Maintenance  Actions  per  Flight  Hour  Time  Series  for  Active 

Duty  Marine  Corps  Squadrons. 

Each  panel  represents  28  months  of  data  from  January  2003  to  April  2005.  Months 
during  which  squadrons  flew  fewer  than  50  flight  hours  have  been  omitted. 

Most  squadrons  exhibit  noticeable  variability  with  respect  to  this  metric.  Some 
squadrons,  such  as  VMFA121,  122,  225,  232  and  251  show  increasing  values  of 
this  metric,  which  is  an  undesirable  trend.  Others,  such  as  VMFA312,  show  a 
movement  in  the  desired  direction.  There  is  no  single  obvious  characteristic  that 
differentiates  these  particular  squadrons,  which  immediately  highlights  the  need 
for  more  techniques  that  can  consider  multiple  variables. 

d.  Cannibalizations  per  Flight  Hour 

Another  metric  that  may  provide  an  indication  of  maintenance 
performance  is  the  cannibalization  rate,  expressed  as  cannibalizations  per  flight 
hour.  Squadrons  strive  for  lower  values  of  this  indicator,  since  higher  values 


reflect  a  less-responsive  supply  system  and  an  increase  in  the  man-hours 
required  to  achieve  a  desired  level  of  output  (flight  hours,  mission  capable 
aircraft,  etc).  Figure  6  depicts  the  cannibalizations  per  flight  hour  time  series  with 
least-squares  trend  lines  superimposed. 
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Figure  6  Cannibalizations  per  Flight  Hour  Time  Series  for  Active  Duty  Marine 
Corps  F/A-18  Squadrons 

Each  panel  represents  31  months  of  observations  for  a  given  squadron.  Least-squares 
trend  lines  are  superimposed  on  each  squadron’s  time  series. 


VMFA121  shows  upward  movement  in  the  cannibalizations  per  flight  hour  metric, 
which  is  an  undesirable  trend,  whereas  VMFA314  shows  desirable  downward 
trending.  High  values  of  this  metric  may  reflect  supply  shortfalls,  lower  personnel 
experience  levels,  training  deficiencies,  or  mismanagement  of  resources. 

e.  Technical  Directive  (TD)  Hours 
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As  described  in  Chapter  II,  TD’s  are  specialized  maintenance 
actions,  directed  by  Naval  Air  Systems  Command  (NAVAIR),  which  can  be  of  an 
immediate  or  less  urgent  nature.  Squadrons  are  obligated  to  incorporate  those 
TDs  that  affect  flight  safety  by  performing  immediate,  dedicated  repairs,  but 
squadrons  tend  to  address  those  of  a  less  urgent  nature  when  the  aircraft  are 
undergoing  other  preventive  or  corrective  maintenance.  Some  squadrons  may 
classify  TDs  as  unscheduled  maintenance  and  will  therefore  accrue  NMCMU 
time  [reference  Chris  Hawes  email].  To  the  extent  that  this  is  the  case,  TDs  may 
be  correlated  with  the  NMCMU  metric.  Figure  7  overlays  the  hours  required  to 
incorporate  these  TDs  and  the  NMCMU  metric  on  one  plot. 


Figure  7  NMCMU  and  TD  Hours  by  Month  by  Squadron  Time  Series 

For  some  squadrons,  such  as  VMFA225  and  VMFA332,  there 
appears  to  be  related  movement  between  the  TD  hours  and  NMCMU  metrics.  To 
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a  certain  extent,  therefore,  we  may  be  describing  the  same  performance  issue 
with  two  different  metrics.  Without  considering  the  number  and  type  of  the  TDs 
scheduled,  TD  hours  reflect  the  type/model/series  aircraft  requiring  them  more  so 
than  the  performance  of  the  maintainers.  Since  the  squadrons  are  not  usually 
under  immediate  pressure  to  complete  TDs,  the  month  in  which  they  are 
scheduled,  started,  and  completed  is  not  captured  in  sufficient  detail  to  develop  a 
better  metric.  A  more  detailed  analysis  is  required  to  quantify  the  portion  of 
maintenance  that  is  directly  attributable  to  the  incorporation  of  TDs,  normalized 
by  the  total  number  of  TDs  scheduled  for  action. 

2.  Descriptive  Metrics 

With  performance  measures  identified,  we  next  identify  a  set  of  predictor 
variables  to  statistically  explain  these  measures.  From  this  point  forward,  we  limit 
our  focus  to  man-hours  per  maintenance  action  as  our  dependant  variable,  to 
demonstrate  the  viability  of  our  analytical  approach.  We  examine  the  predictor 
variables  within  broad  categories  of  operational,  personnel,  inventory,  and 
technical  support  metrics. 

We  can  quickly  check  for  correlation  between  the  response  variable  man¬ 
hours  per  maintenance  action  and  our  predictor  variables  with  pairwise 
scatterplots,  which  are  shown  in  Appendix  D.  The  plots  do  not  indicate  any 
obvious  correlation  between  the  response  variable  and  the  predictors.  This  plot 
also  provides  an  opportunity  to  eliminate  redundant  predictor  variables.  None  of 
the  scatterplots  cause  us  to  eliminate  variables  at  this  point, 
a.  Operational  Metrics 

Deployment  status.  As  described  in  Chapter  II,  deployment  status 
categorizes  the  overall  operational  context  of  a  squadron  at  a  moment  in  time 
into  one  of  four  levels:  “IRAQ,”  “UDP,”  “CVN,”  and  “CONUS.”  “IRAQ”  identifies 
those  high-priority  deployments  such  as  those  in  support  of  Operation  Enduring 
Freedom  and  Operation  Iraqi  Freedom.  These  particular  operations  were 
supported  by  both  land-based  and  carrier-based  aircraft  during  February  -  May 
2003,  and  again  in  the  months  following  September  2004.  As  indicated  in 
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Figure  8  ,  the  land-based  squadrons  that  participated  in  the  2003  operations — 
VMFA-121,  VMFA-225,  VMFA-225,  and  VMFA-533 — flew  nearly  three  times  their 
normal  monthly  flight  hours,  and  carrier-based  squadrons  exhibit  spikes  in  flight 
hours  as  well. 
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Figure  8  Flight  Hours  per  Month  and  Deployment  Status  for  Marine  Corps 

F/A-18  Squadrons. 

Each  panel  represents  a  separate  squadron’s  set  of  observations  for  the  31  months 
between  October  2002  and  April  2005.  The  deployment  status  categorical  variable 
has  four  levels: 

“Iraq,”  which  indicates  a  combat  deployment  in  support  of  Operation  Iraqi  Freedom 
“CVW,”  which  indicates  a  deployment  with  a  carrier  air  wing 
“UDP,”  which  indicates  a  deployment  with  the  Unit  Deployment  Program 
“CONUS,”  which  indicates  that  the  squadron  is  operating  within  the  U.S. 

Unlabeled  portions  of  the  deployment  status  plot  are  of  the  CONUS  level. 
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Other  levels  of  this  metric  include  “CVN”  and  “UDP,”  which  indicate  the  unit’s 
participation  in  a  carrier  deployment  or  Unit  Deployment  Program  deployment 
(UDP),  respectively.  The  UDP  program  rotates  squadrons  to  bases  in  the 
Western  Pacific  Theater  of  Operations  for  six-month  deployments  at  regularly 
scheduled  intervals.  Since  the  deployment  metric  moves  in  discrete  jumps  every 
few  months,  it  can  not  explain  month-to-month  variations  in  performance. 
However,  it  may  allow  us  to  explain  some  of  the  variance  in  performance  by 
representing  those  characteristics  that  are  not  explicitly  included  -  morale, 
urgency,  and  prioritization. 

b.  Personnel  Metrics 

Intuitively,  the  performance  of  a  squadron  is  related  to  the  quality  of 
personnel  conducting  its  work.  Our  challenge  is  to  identify  metrics  that  capture 
attributes  of  personnel  quality.  If  we  consider  the  workforce  as  a  dynamic  entity 
that  accumulates  knowledge  and  experience  over  time,  then  the  quality  of  a 
squadron  workforce  might  be  expressed  as  a  sum  of  these  factors.  Likewise,  with 
the  loss  of  experienced  personnel  comes  a  loss  in  the  aggregate  experience  of 
the  squadron.  We  use  three  metrics  to  capture  this  phenomenon:  months  of 
service,  months  in  squadron,  and  turnover. 

Months  of  Service.  Again,  we  expect  to  see  a  relationship 
between  the  capability  of  the  workforce  and  the  quality  of  work  it  produces. 
Capability,  while  not  directly  measurable,  may  be  reflected  in  the  experience  of 
the  squadron  maintainers.  We  first  define  experience  as  months  of  service,  which 
indicates  the  total  number  of  months  the  maintainer  has  been  on  active  duty. 
From  personnel  data,  the  individuals’  months  of  service  is  noted  at  the  end  of 
each  month.  Figure  9  shows  histograms  of  months  of  service  for  the  maintainers 
of  the  squadrons  in  our  study.  It  is  clear  that  the  distribution  of  experience  is  not 
symmetrical. 
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Figure  9  Distribution  of  Months  of  Experience  of  Squadron  Maintainers. 

Each  panel  represents  twenty  months  of  data,  from  May  2003  to  December  2004,  for 
each  of  the  squadrons  in  our  study.  Data  represents  maintenance  personnel  only,  and 
was  not  obtained  for  VMFA-121.  Squadron  labels  are  located  above  each  histogram. 
Horizontal  axis  labels  represent  months  active  service;  vertical  axes  represent  numbers 
of  personnel. 


The  fact  that  all  squadrons  exhibit  a  skewed  distribution  of  months  of  service 

suggests  that  most  individuals’  experience  levels  lie  below  the  squadron  mean, 

which  ranges  from  62.4  months  of  service  (VMFA533)  to  73.3  months  of  service 

(VMFA251).  Some  squadrons  do  appear  to  have  a  wider  spread  of  experience 

levels  than  others,  which  may  be  even  more  pronounced  when  viewed  across 

time.  We  use  the  first  quartile,  second  quartile  (median),  and  third  quartile 

together  to  capture  these  shape  characteristics.  For  a  given  month,  the  value  of 

the  first  quartile  indicates  that  25  percent  of  the  maintainers  in  the  squadron  have 
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that  many  or  fewer  months  on  active  duty.  These  quartiles  are  plotted  as  a  time 
series  in  Figure  10  .  The  plots  suggest  that  the  second  quartile  may  contain 
redundant  information,  so  we  will  include  only  the  first  and  third  quartiles  when 
considering  them  for  model  inclusion. 
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Figure  10  Months  of  Service  Quartiles  Time  Series. 

Each  panel  represents  20  months  of  the  months  of  service  quartiles  from  May 
2003  to  December  2005  for  active  duty  Marine  Corps  F/A-18  squadrons.  Data  for 
VMFA-121  was  not  obtained. 

As  Figure  10  shows,  the  time  series  exhibit  significant  movement 
during  the  period  of  our  study.  VMFA251  appears  to  exhibit  an  increasing 
experience  level  in  the  upper  quartile,  whereas  VMFA232  and  225  show 
declines.  Furthermore,  the  movement  of  the  experience  level  of  the  lower  quartile 
does  not  necessarily  correspond  to  that  of  the  upper  quartile,  as  seen  with 
VMFA1 15  and  VMFA122. 
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Months  in  Squadron.  If  the  average  accrued  time  spent  in  a 
squadron  is  relatively  low,  we  would  expect  performance  to  suffer  to  some 
degree.  Figure  1 1  depicts  the  movement  of  the  months  in  squadron  quartiles 
over  time  for  each  of  the  squadrons. 


Figure  11  Maintainer  Months  in  Squadron  Quartiles  Compared  to  Man-Hours 
per  Maintenance  Action  (MMHperMA) 


During  the  two-year  period  under  investigation,  we  see  significant  drops  in 
squadron  experience  for  some  squadrons,  and  increases  for  others;  again,  the 
upper  and  lower  quartiles  do  not  necessarily  correspond.  When  viewed  in 
relation  to  the  man-hours  performance  metric,  we  don’t  see  obvious  correlation, 
so  it  is  difficult  to  prefer  one  particular  quartile  over  another  at  this  stage. 
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Turnover.  In  addition  to  varying  experience  levels  of  the  personnel 
in  the  squadron,  we  know  that  squadrons  experience  a  certain  amount  of 
turnover — a  function  of  people  entering  and  leaving  the  organization  in  any  given 
month.  Figure  12  depicts  the  turnover  quantities  by  month,  by  squadron. 
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Figure  12  Maintenance  Personnel  Movement  for  Marine  Corps  F/A-18 
Squadrons. 

The  panels  represent  personnel  data  for  the  20  months  between  May  2003  and 
December  2004.  Data  for  VMFA-121  was  not  obtained.  The  grey  bars  above  the 
horizontal  line  show  the  number  of  maintainers  that  entered  the  squadron  during 
the  corresponding  month,  on  a  scale  from  0  to  60.  Similarly,  the  bars  below  the 
horizon  line  indicate  the  number  of  maintainers  that  departed,  on  a  scale  from  0 
to  -60.  The  total  number  of  maintainers  in  the  squadron  is  shown  as  well,  on  a 
scale  from  0  to  200. 


We  can  see  significant  quantities  of  inbound  and  outbound 
personnel  at  specific  times  during  the  period  under  investigation.  We  capture  the 

turnover  information  in  a  single  metric  in  which  we  calculate  the  total  number  of 
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inbound  and  outbound  maintenance  personnel  as  a  percentage  of  the  total.  We 
might  expect  units  with  especially  high  turnover  rates  to  struggle  with 
maintenance  performance.  Figure  13  depicts  the  turnover  for  each  squadron  as  a 
time  series. 
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Figure  13  Maintenance  Personnel  Turnover  by  Squadron  Time  Series 
Compared  to  Man-hours  per  Maintenance  Action  (MMHperMA). 

Each  panel  represents  observations  from  an  individual  Marine  Corps  F/A-18 
squadron.  Squadron  labels  are  above  each  plot.  Turnover  is  defined  as  the  total 
number  of  maintenance  personnel  into  and  out  of  the  squadron  as  a  proportion  of  the 
total  number  of  maintenance  personnel.  The  vertical  axis  for  turnover  ranges  from  0 
to  0.5. 


It  is  apparent  that  the  squadrons  occasionally  experience  relatively 
high  levels  of  turnover,  such  as  VMFA314  and  VMFA323.  We  might  expect  the 
squadrons  to  benefit  from  a  stable  manpower  base  and  likewise  to  suffer  from  an 


environment  of  high  personnel  turnover.  With  respect  to  our  performance  metric, 
man-hours  per  maintenance  action,  the  time  series  plot  shows  little  direct 
correlation  to  turnover. 

c.  Inventory  Metrics 

We  might  reasonably  expect  older  aircraft  to  exhibit  higher  failure 
rates  and  require  additional  maintenance  when  compared  to  newer  aircraft.  The 
inventory  of  F/A-18  aircraft  operated  by  the  squadrons  under  investigation  differ 
significantly  in  their  age  in  terms  of  months  of  service  and  accrued  hours  flown. 
Figure  14  depicts  time  series  of  box  plots  that  show  the  distribution  of  accrued 
hours  on  the  squadrons’  inventory  of  aircraft.  The  oldest  lots  of  aircraft  included 
in  this  data  came  off  the  production  line  in  1986;  the  newer  aircraft,  by  contrast, 
were  accepted  by  the  operating  forces  in  2000.  We  can  see  that  the  older  aircraft 
have  nearly  twice  the  accrued  flight  hours  as  that  of  those  that  operate  newer  lots 
of  aircraft.  When  viewed  as  a  time  series,  we  see  the  average  ages  of  aircraft  in 
each  squadron  slowly  increase  as  we  might  expect.  We  also  see  sudden  rises 
and  falls  of  the  averages,  attributable  to  those  points  in  time  where  squadron 
exchanged  aircraft  for  operational  and  service  life  extension  reasons.  These 
particular  points  of  aircraft  exchange  may  themselves  be  an  explanatory  factor  to 
a  performance  measure. 
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Figure  14  Boxplots  of  Airframe  Hours  by  Squadron. 

Each  panel  represents  a  time  series  of  boxplots  for  the  24  months  between  May 
2003  and  April  2005.  Each  squadron  is  normally  assigned  twelve  aircraft; 
therefore,  each  boxplot  represents  the  age  distribution  (in  hours  flown)  of  the 
squadron’s  inventory  of  aircraft,  for  the  corresponding  month.  The  box  represents 
all  data  between  the  25th  and  75th  percentiles.  The  line  inside  the  box  represents 
the  median  of  the  distribution.  The  vertical  lines  that  extend  above  and  below  the 
box  represent  the  range  of  data;  horizontal  tick  marks  outside  these  ranges 
represent  outliers. 

We  reduce  the  aircraft  age  distributions  to  a  single  metric,  average 
aircraft  hours  in  service,  and  plot  the  time  series  of  this  average  along  with  man¬ 
hours  per  maintenance  action  in  Figure  15  .  The  plot  does  not  suggest  an 
obvious  relationship  between  the  aircraft  age  and  our  performance  metric  man¬ 
hours  per  maintenance  action. 
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Figure  15  Mean  Aircraft  Hours  in  Service  Compared  to  Man-Hours  per 
Maintenance  Action  (MMHperMA). 

Each  panel  represents  28  months  of  the  man-hours  per  maintenance  action  metric, 
for  the  months  between  January  2003  and  April  2005,  for  Marine  Corps  F/A-18 
squadrons.  MMHperMA  is  shown  on  a  scale  ranging  from  0  to  8.  Average  aircraft 
hours  in  service  metric  is  shown  on  a  scale  ranging  from  1500  to  6000  hours. 

d.  Engineering  and  Technical  Support  (ETS)  Metrics 

The  entry  screen  for  new  ELAR  records  is  shown  in  Appendix  B. 
Our  goal  is  to  identify  metrics  related  to  tech  rep  utilization  that  may  exhibit 
relationship  with  performance  measures  at  the  squadron  level.  We  simplify  the 
search  by  eliminating  ELAR  data  fields  that  pertain  to  subjective  comments, 
descriptive  text,  and  freeform  feedback. 

Each  time  a  customer  initiates  a  request  for  ETS  support,  a  new 
ELAR  record  is  generated.  Theoretically,  the  demand  for  ETS  support,  reflected 
by  the  number  of  tech  rep  requests  received,  corresponds  with  the  number  of 
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records  in  the  database.  During  the  period  August  2003  to  April  2005,  ELAR 
contains  6,249  records  for  which  the  program  is  categorized  as  “F-18.”  Not  all 
squadrons  use  ELAR  with  the  same  regularity,  as  shown  in  Figure  16  .  We  are 
interested  in  records  that  can  be  attributed  to  a  particular  squadron.  However,  of 
these  6,249  F/A-18  records,  only  3,176  of  these  (51  percent)  attribute  the  tech 
rep  action  to  a  specific  squadron.  The  situation  is  improving:  after  January  2005, 
the  squadron  data  field  is  almost  always  present.  This  improvement 
notwithstanding,  we  are  left  with  a  very  short  time  frame  in  which  to  study  the 
effects  of  tech  rep  actions  on  squadron  performance,  using  reliable  data. 
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Figure  16  Distribution  of  Monthly  ELAR  Records. 

Panels  show  the  time-series  distribution  of  ELAR  records  between  January  2004  and 
June  2005. 
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The  distribution  shows  a  sharp  decline  in  the  number  of  records  during  July  and 
August  2004.  Either  ETS  support  decreased  during  this  time  period  or  the  tech 
reps’  documentation  in  ELAR  diminished  for  some  other  reason  unknown  to  us. 

We  expect  the  volume  of  ETS  requests  to  vary  by  type  of  support 
offered.  An  important  role  of  the  tech  reps  is  to  provide  specific  training  that 
augments  the  basic  skills  training  of  the  maintainers  obtained  after  they  complete 
recruit  training  and  prior  to  their  arrival  at  their  first  unit.  Such  skill  training  is 
usually  unique  to  a  specific  work  center  and  is  conducted  during  formal  and 
informal  training  in  short  periods  as  the  schedule  permits.  Since  the  conduct  of 
this  training  is  not  standardized  in  its  conduct  or  in  its  documentation,  we  do  not 
quantify  it  directly  in  this  study.  However,  the  tech  reps  themselves  conduct  a 
portion  of  this  training  and  document  such  activities  as  formal  training  and  on  the 
job  training  in  their  ELAR  database. 
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Percentages  of  Records  by  Problem  Type 
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Figure  17  Pareto  Chart  Distribution  of  F/A-18  ELAR  Records  by  Problem  Type 

The  chart  includes  the  6249  ELAR  records  recorded  between  August  2003  and 
April  2005.  The  most  common  problem  types — troubleshooting,  on-the-job- 
training,  formal  training,  and  research — account  for  nearly  90  percent  of  all 
records.  The  vertical  axis  is  a  percentage  scale. 
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In  addition  to  records  that  are  associated  with  training  squadron 
personnel,  we  can  identify  any  other  ETS  metrics  that  are  most  often 
documented  by  the  tech  reps  and  the  requesting  units.  Figure  17  depicts  a 
Pareto  plot  of  the  distribution  of  records  by  problem  type.  The  first  four  categories 
of  problem  type — troubleshooting,  on-the-job-training  (OJT),  formal  training,  and 
research — account  for  nearly  90  percent  of  F/A-18  ELAR  records;  we  therefore 
limit  consideration  to  these  problem  types.  We  sum  the  monthly  records  into  a 
single  ELAR  metric  and  plot  the  changing  values  as  a  time  series,  as  shown  in 
Figure  18  ,  compared  to  the  performance  metric  man-hours  per  maintenance 
action. 


VMFA121 


VMFA122 


VMFA323 


~  ~  ~ 

VMFA332 


Man-Hours  per  Maintenance  Action 
ELAR  Records 


Oct02  Apr03  Oct03  Apr04  Oct04  Apr05 


Figure  18  ELAR  Records  Compared  to  Man-Hours  per  Maintenance  Action. 

Each  panel  represents  data  for  a  Marine  Corps  F/A-18  squadron.  ELAR  records  are 
limited  to  the  21  months  between  August  2003  and  April  2005  and  to  records  that 
can  be  attributed  to  a  specific  squadron. 


Figure  18  indicates  that  some  squadrons,  such  as  VMFA-115,  VMFA-232,  and 
VMFA-314,  report  much  higher  numbers  of  records  on  average;  it  is  possible  that 
these  squadrons  have  different  policies  or  procedures  regarding  their  compliance 
with  ELAR.  To  build  an  accurate  explanatory  model  that  includes  ELAR  record 
counts,  we  would  need  to  explore  records  with  the  squadron  field  missing  in 
detail,  which  is  beyond  the  scope  of  this  research.  Chapter  III.E.  outlines  the 
approach  we  take  when  analyzing  the  power  of  ELAR  activity  in  predicting 
performance  variability.  Chapter  IV  discusses  improvements  in  the  ELAR 
database  tool  that  could  improve  analysts’  ability  to  measure  tech  rep  activity 
accurately. 

e.  Location 

Each  squadron  is  associated  with  a  particular  location,  or  home 
station,  as  listed  in  Table  1.  We  are  interested  in  knowing  whether  some  of  the 
variability  of  man-hours  per  maintenance  action  can  be  explained  by  the  location 
of  the  squadron.  We  create  a  two-level  categorical  variable  location  to  capture 
this  factor.  Figure  19  shows  the  distribution  of  man-hours  per  maintenance  action 
for  the  two  levels  of  location. 
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Figure  19  Boxplots  of  Man-Hours  per  Maintenance  Action  by  Location. 

Each  boxplot  shows  the  distribution  of  man-hours  per  flight  hour  for  the  31 
months  of  data  between  October  2002  and  April  2005.  The  two  boxplots  are  for 
Marine  Corps  F/A-18  squadrons  based  at  MCAS  Beaufort  and  MCAS  Miramar, 
respectively.  The  box  represents  all  data  between  the  25th  and  75th  percentiles. 
The  line  inside  the  box  represents  the  median  of  the  distribution.  The  vertical 
lines  that  extend  above  and  below  the  box  represent  the  range  of  data;  dots 
outside  these  ranges  represent  outliers. 


Figure  19  indicates  that  there  is  a  small  difference  between  the  distributions  of 
Beaufort  and  Miramar  observations  with  respect  to  man-hours  per  maintenance 
action. 

f.  Operational  Metrics 

As  discussed  in  Section  C.2.  of  this  chapter,  the  Marine  Corps 
executes  a  cyclical  readiness  policy  whereby  units  that  are  nearing  deployment 
enjoy  a  focus  of  effort  and  maintain  a  high  state  of  readiness.  Conversely, 
returning  units  receive  lower  priority  and  may  expect  to  achieve  a  lower  state  of 
readiness.  Factors  such  as  personnel  morale  and  sense  of  urgency  are  also 

affected  by  the  operational  status  of  the  squadron.  Since  we  are  aware  of  these 
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phenomena,  we  consider  deployment  status  as  an  explanatory  factor  in  the 
variability  of  man-hours  per  maintenance  action.  Figure  20  shows  a  time  series 
plot  of  both  the  performance  measure  man-hours  per  maintenance  action  and 
deployment  status.  The  plots  alone  do  not  suggest  a  direct  relationship  between 
the  two  metrics. 
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Figure  20  Deployment  Status  Compared  to  Man-Hours  per  Maintenance 
Action  and  Deployment  Status. 


The  man-hours  performance  metric  does  not  appear  to  be  affected  by 
deployment  status.  We  might  expect  operational  deployments  to  result  in  an 
increased  performance  level,  reflected  in  fewer  man-hours  per  maintenance 
action;  however,  we  can  not  draw  this  conclusion  from  the  plots  alone. 
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g.  Summary  of  Exploratory  Analysis 

We  have  compiled  several  variables  that  we  believe  may  be 
significant  in  explaining  variability  of  squadron  maintenance  performance.  These 
variables  are  listed  in  Table  3. 


Variables 

Source 

Data  Fields  Used 

Data  Range 

Man  Hours  per  Maint.  Action 

NALCOMIS 

man-hours,  maintenance  actions 

Oct  02  -  Apr  05 

Months  of  Service  Quartiles 

MCTFS 

months  active  service,  months  active 

service 

May  03  -  Dec  04 

May  03  -  Dec  04 

Months  in  Squadron  Quartiles 

MCTFS 

date  of  record,  date  arrived  duty  station 

May  03  -  Dec  04 

Turnover 

MCTFS 

arrived  date,  record  date 

May  03  -  Dec  04 

Average  Aircraft  Hours  in  Service 

NAVAIR 

hours  in  service 

May  03  -  Apr  05 

ELAR  Records 

ELAR 

N/A 

Aug  03  -  Apr  05 

Deployment 

Various 

N/A 

Oct  02  -  Apr  05 

Flight  Hours 

NALCOMIS 

flight  hours 

Oct  02  -  Apr  05 

Location 

N/A 

N/A 

Oct  02  -  Apr  05 

Type  Equipment  Code 

NALCOMIS 

type  equipment  code  (TEC) 

Oct  02  -  Apr  05 

Organization 

N/A 

N/A 

Oct  02  -  Apr  05 

Table  3.  Table  of  Potential  Predictor  and  Response  Variables. 

Data  used  in  the  calculation  of  the  variables  in  the  table  are  calculated  on  a  monthly  basis.  For 
personnel  data,  months  in  squadron  is  determined  by  subtracting  the  month  the  individual  arrived 
at  the  duty  station  from  the  current  month  of  record.  Turnover  is  calculated  by  summing  inbound 
and  outbound  personnel  and  dividing  by  the  total  number  of  maintenance  personnel  in  the 
squadron.  Inbound  personnel  are  counted  by  summing  those  records,  in  the  given  month,  for 
which  arrived  date  equals  the  date  of  the  record. 

In  the  next  section  we  explain  the  model  selection  and  estimation  process,  using 
the  performance  measure  man  hours  per  maintenance  action  as  the  response 
variable  and  the  other  variables  of  Table  3  as  predictor  variables. 
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D.  PREDICTOR  VARIABLE  CORRELATION 

Before  beginning  model  selection  process,  we  examine  the  set  of 
predictor  variables  for  indications  of  redundancy.  Figure  21  shows  a  scatterplot  of 
variables  pertaining  to  personnel  experience. 
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Figure  21  Personnel  Experience  Metric  Pairwise  Scatterplots. 

Each  panel  represents  240  observations:  20  months  of  data  between  August  2003 
and  December  2004  for  12  active  duty  Marine  Corps  F/A-18  squadrons.  Personnel 
data  from  VMFA-121  is  not  included. 

The  pair-wise  scatter  plots  suggest  fairly  high  levels  of  correlation  between  these 
metrics.  The  correlation  value  between  months  in  squadron,  first  quartile 
(TimelnSqdnQ25)  and  months  in  service,  first  quartile  (MoExpIstQ),  is  0.81.  The 
use  of  highly  correlated  predictor  variables  increases  estimation  error,  and 
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makes  it  difficult  to  attribute  effects  to  particular  variables.  Pair-wise  scatter  plots 
of  the  remaining  metrics,  not  including  the  categorical  variables  TEC, 
deployment,  location  or  organization,  can  be  found  in  Appendix  D. 

E.  MODEL  BUILDING 

With  potential  areas  of  multi-collinearity  identified,  we  turn  to  the  problem 
of  determining  the  best  combination  of  the  predictors  that  explain  our  chosen 
response  variable,  man-hours  per  maintenance  action.  For  the  analysis,  we  use 
an  ordinary  least  squares  (OLS)  linear  regression  to  develop  an  explanatory 
model.  The  data  set  we  use  contains  403  observations  of  1 1  predictor  variables. 
However,  since  some  observations  contain  missing  values,  the  actual  number  of 
observations  used  in  model  estimation  depends  on  which  variables  are  included. 
When  including  all  eleven  predictors,  we  have  20  months  of  complete  data  for  12 
of  the  13  Marine  Corps  F/A-18  squadrons.  Several  months  of  the  personnel 
variables  are  thought  to  be  erroneous  and  are  omitted,  leaving  a  total  of  209 
observations  with  no  missing  values. 

Each  of  the  k  levels  of  the  categorical  variables — TEC,  location, 
deployment,  and  organization — are  automatically  assigned  k- 1  “dummy” 
variables  by  the  statistics  software.  For  our  analysis,  we  use  S-Plus®  version  6.2 
[Insightful  Corporation,  2003],  Table  4.  lists  the  variable  abbreviations  shown  in 
S-Plus  reports: 


S-Plus  Abbreviation 

Variable 

TECAMAA 

Type  equipment  code  =  “AMAA”  (F/A-18A) 

TECAMAF 

Type  equipment  code  =  “AMAF”  (F/A-18C) 

TECAMAG 

Type  equipment  code  =  “AMAG”  (F/A-18D) 

AvgAircraftHrsInSvc 

Average  aircraft  hours  in  service 

MoExpl  stQ/2ndQ/MoExp3rdQ 

Months  in  service,  first  quartile/second  quartile/third  quartile 

TimelnSqdnQ25/Med/Q75 

Months  in  squadron,  first  quartile/second  quartile/third  quartile 

Flthrs 

Flight  hours 

Loc 

Location  =  “Miramar” 

Table  4.  Variable  Coding  and  Abbreviations  in  S-Plus  Reports. 
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We  know  that  the  squadrons’  performance  levels  differ  during  any  given 
month,  but  our  goal  is  to  identify  those  characteristics  that  differentiate  the 
squadrons  in  this  respect.  We  therefore  begin  by  building  a  least-squares 
regression  model  without  the  organization  term  to  see  how  much  of  the  variability 
can  be  explained  by  underlying  squadron  characteristics.  Montgomery,  Peck, 
and  Vining  [2001]  give  a  thorough  explanation  of  linear  modeling  using  least 
squares  regression.  Finally,  we  address  the  first  three  study  questions  posed  in 
Chapter  I: 

1.  Which  squadron  characteristics  have  a  detectable  contribution  to 
the  variability  of  the  performance  measure  man-hours  per  maintenance 
action ? 

2.  How  much  additional  variability  is  explained  by  the  squadron  that  is 
not  accounted  for  by  the  squadron  characteristics  already  considered? 

3.  Is  there  a  time-of-year  effect  for  the  performance  of  the  squadrons? 

We  begin  by  identifying  a  statistical  model  that  will  apply  to  all  squadrons 

similarly,  using  measurable  characteristics  of  the  squadrons.  We  then  compare 
this  model  to  one  that  includes  the  organization  term,  which  captures  the 
squadrons  directly.  Although  the  latter  model  may  have  better  explanatory  power 
than  the  former  model,  it  lends  less  insight  into  why  a  particular  squadron  may 
perform  differently  from  another.  By  using  squadron  characteristics  to  capture 
this  effect,  we  explain  maintenance  performance  in  a  manner  that  is  applicable 
beyond  the  thirteen  squadrons  that  were  included  in  our  research. 

1.  Full  Models 

We  begin  our  modeling  effort  by  identifying  the  linear  combination  of 
predictor  variables  that  best  explains  or  predicts  man  hours  per  maintenance 
action.  Using  the  linear  regression  utility  in  S-Plus,  the  initial  full  model  includes 
all  potential  predictors  as  main  effects  variables.  However,  rather  than  include 
organization  in  the  model,  we  proceed  with  the  regression  without  an 
organization  term  and  then  analyze  the  distribution  of  residuals  by  organization. 
This  process  will  allow  us  to  analyze  the  added  explanatory  power  of  the  added 
organization  term. 
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We  express  the  response  variable  as  a  linear  combination  of  predictor 
variables  plus  an  error  term.  Formally, 


Ys,t  ~  00  +  P\X\ ,s,t  +  PlX2 ,s,t  +  PiX3 ,s,t  +  PaX 4,s,t  +  PsX5,s,t  + 

PbXb,s,t  +  PlX7,s,l  +  PgXS ,s,l  +  PoX9,s,t  +  PlOX  10, s,t  +  PnXl\ ,s,t  +  £s, 


(1) 


where 


7, ,  =  man-hours  per  maintenance  action,  squadron  s,  month  t 
X\  5j  =  type  equipment  code 
X2  s ,  =  first  quartile,  months  experience 
X2  u  =  third  quartile,  months  experience 
X4sl=  average  aircraft  hours  in  service 
XSst  =  turnover 
X6sl=  location 

Xlsl  =  first  quartile,  months  in  squadron 
XSsl  =  second  quartile,  months  in  squadron 
X9sl  =  third  quartile,  months  in  squadron 
Xl0  s ,  =  deployment 
Xn  s,  =  flight  hours 
ss ,  =  residual 
k=  number  of  variables 
s=  squadron 
t=  month 


We  have  included  the  full  set  of  predictors  in  Equation  (1).  For  this  model, 
we  have  209  observations  with  no  missing  values  of  these  variables.  The  eleven 
predictors  and  the  additional  levels  needed  for  the  categorical  variables 
constitute  14  regression  variables  plus  an  intercept  term.  The  plot  of  the 
residuals  against  the  fitted  values  indicates  non-constant  variance  of  the 
residuals.  The  plot  of  the  residuals  against  the  fitted  values  suggests  that 
variance  of  the  residuals  is  not  independent  of  the  predicted  values.  The  normal 
plot  of  residuals  also  shows  some  skewing.  We  therefore  transform  the  response 
variable  with  a  natural  logarithm  transformation.  After  transforming  the  response 
variable  accordingly  we  are  left  with  a  full  model  expressed  formally: 


ln  Ys,t  -  A)  +  P\X\,s,t  +  PlX2,s,t  +  PiX3,S,t  +  04X4 ,s,t  +  PsX5, s,t  + 

PbX6 ,s,t  +  PlXl,s,t  +  P$XS,s,t  X  PoX9 ,s,t  X  Pl0X10,s,t  +  PnXU,s,t  X  Ss. 


(2) 
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The  output  from  this  full  regression  model  in  S-Plus  is  shown  in  Appendix 
E.  The  summary  data  from  this  regression  is  reproduced  in  Figure  22  . 


Coefficients  : 


Value  Std.  Error  t  value  Pr(>|t|) 


( Intercept) 

1 

.2280 

0 

.2453 

5 

.  0060 

0 

.0000 

TECAMAF 

0 

.0517 

0 

.0708 

0 

.  7304 

0 

.4660 

TECAMAG 

0 

.1865 

0 

.0816 

2 

.2861 

0 

.  0233 

MoExpIstQ 

0 

.0011 

0 

.0071 

0 

.  1620 

0 

.8715 

MoExp3rdQ 

-0 

.0026 

0 

.0022 

-1 

.  1800 

0 

.2395 

AvgAircraftHrsInSvc 

0 

.0001 

0 

.0000 

1 

.  7798 

0 

.0767 

Turnover 

-0 

.2300 

0 

.2191 

-1 

.  0497 

0 

.2952 

Loc 

0 

.2214 

0 

.0389 

5 

.6884 

0 

.0000 

TimeInSqdnQ25 

0 

.0005 

0 

.0065 

0 

.  0815 

0 

.9351 

TimelnSqdnMed 

-0 

.0179 

0 

.0057 

-3 

.  1402 

0 

.0020 

TimeInSqdnQ75 

-0 

.0090 

0 

.0071 

-1 

.2650 

0 

.2074 

DeploymentUDP 

-0 

.1124 

0 

.0512 

-2 

.  1929 

0 

.  0295 

DeploymentCVN 

0 

.1827 

0 

.1099 

1 

.  6634 

0 

.0978 

Deployment IRAQ 

0 

.0680 

0 

.2092 

0 

.3249 

0 

.7456 

Flthrs 

0 

.0000 

0 

.0001 

0 

.2025 

0 

.8397 

MoExp2ndQ 

0 

.0064 

0 

.0072 

0 

.  8896 

0 

.3748 

Residual  standard  error:  0.212  on  194  degrees  of  freedom 
Multiple  R-Squared:  0.2791 

F-statistic:  5.007  on  15  and  194  degrees  of  freedom,  the  p-value  is  2.751e-008 
194  observations  deleted  due  to  missing  values 

Figure  22  Full  Model  Summary,  S-Plus  Report. 

The  model  includes  all  predictors.  The  available  degrees  of  freedom  are  based  on  the 
number  of  observations  n  =  209.  p  =  15  (14  variables  plus  the  intercept)  leaving  194 
degrees  of  freedom.  R2  is  derived  by  dividing  error  sums  of  squares  by  total  sums  of 
squares  and  subtracting  from  1.  The  F-statistic  is  based  on  p-1  =  14  and  n-p=225  degrees 
of  freedom.  Variables  are  abbreviated  in  S-Plus.  TEC  =  type  equipment  code ; 
MoExpIstQ  =  months  in  service,  first  quartile;  AvgAircraftHrsInSvc  =  average  aircraft 
hours  in  service:  TimelnSqdn  =  months  in  squadron:  Loc  =  location:  Flthrs  =  flight  hours. 


An  initial  indication  of  the  model’s  ability  to  explain  the  variance  is  seen  in 
the  R2  value.  The  R2  value  is  calculated  by  dividing  the  error  sums  of  squares  by 
the  total  sums  of  squares  and  subtracting  from  1 .  As  seen  in  Figure  22,  R2  for  the 
full  model  is  0.279.  This  R2  suggests  that  approximately  28  percent  of  the 
variability  is  explained  by  this  model.  However,  not  all  individual  variables  are 
significant  in  the  presence  of  the  others,  as  indicated  by  their  p-values.  Months  in 
squadron  (first  and  third  quartiles),  months  in  squadron  (first  and  third  quartiles), 
and  turnover  appear  to  be  insignificant,  at  the  5%  level,  in  the  presence  of  the 
other  variables  in  this  model. 

2.  Significant  Variables  and  Model  Reduction 
We  proceed  with  stepwise  regression  to  reduce  the  model  to  the  smallest 
model  that  retains  significant  terms.  This  is  implemented  in  S-Plus  software 
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through  the  stepwise  function,  which  uses  Akaike’s  Information  Criterion  (AIC) 
to  determine  the  best  reduced  model  [Insightful,  2001],  The  results  of  the 
stepwise  process  are  shown  in  Appendix  F.  Equation  (3)  expresses  the  reduced 
model. 


In  Ys,t  ~  A)  +  P\X\ ,s,t  +  PlX2,s,,  +  PiX3 ,s,t  +  P\X\ ,s,t  + 
05X5,s,t  ^s,t 


(3) 


Ys  t  =  man-hours  per  maintenance  action,  squadron  s,  month  t 


t  =  type  equipment  code 

X2,s, 

t  =  average  aircraft  hours  in  service 

X3,s, 

t  =  location 

X4,, 

t  =  months  in  squadron,  median 

X5,s, 

t  =  deployment  status 

,  =  residual 

k  =  number  of  variables 
s  =  squadron 
t  =  month 

A  summary  of  the  S-Plus  output  is  shown  in  Error!  Reference  source  not 
found..  The  stepwise  regression  identifies  the  most  significant  terms  with  respect 
to  man-hours  per  maintenance  action:  type  equipment  code,  average  aircraft 
hours  in  service,  location,  months  in  squadron  (median),  and  deployment. 

Since  we  have  transformed  the  response  variable  with  the  natural  log 
function,  this  model  explains  approximately  26.4%  of  the  variability  of  the  natural 
log  of  the  response  variable  ln(7if).  To  find  the  relevant  value  of  R2  that  applies 

to  the  response  variable  directly,  we  convert  the  estimates  of  ln(7s,)  to  those 

representing  Yst  with  the  exponential  function  and  re-calculate  R2;  this  procedure 

results  in  an  R2  of  0.18.  The  F-statistic  indicates  that  the  model  is  significant 
when  compared  to  the  intercept-only  model. 

From  the  coefficients  of  the  regression  model,  we  can  interpret  the 
individual  variable  effects  on  the  natural  logarithm  of  the  performance  metric  man 
hours  per  maintenance  action.  Negative  coefficients  of  numerical  variables 


65 


indicate  that  predicted  values  of  MMHperMA  are  higher  for  the  specified  level  of 
the  variable  than  for  the  level  not  shown.  For  example,  expected  values  of 
MMHperMA  decrease  for  increasing  values  of  TimelnSqdnMed  (median  of 
months  in  squadron),  which  is  the  intuitive  result.  Coefficients  for  categorical 
variables  are  somewhat  harder  to  interpret.  We  use  analysis  of  variance 
(ANOVA)  as  a  way  of  measuring  the  ratio  of  variability  of  a  specific  factor  to  the 
unexplained  variability  (noise);  the  p-values  of  the  ANOVA  table  tell  us  the 
significance  of  the  categorical  term  as  a  whole.  For  example,  the  TEC  term  is 
significant  at  the  0.05  level. 


Call:  lm( formula  =  log (MMHperMA)  ~  TEC  +  AvgAircraftHrsInSvc  +  Loc  + 
TimelnSqdnMed  +  Deployment,  data  =  A,  na. action  =  na. exclude) 
Residuals : 

Min  IQ  Median  3Q  Max 

-0.7744  -0.1301  0.01057  0.1169  0.6243 


Coefficients  : 


Value 

Std 

..  Error 

t  ’ 

value 

Pr 

(>  It  1  ) 

( Intercept) 

1 

.1199 

0  . 

2008 

5 

.5780 

0  . 

.0000 

TECAMAF 

0 

.0281 

0  . 

0670 

0 

.4194 

0 

.  6754 

TECAMAG 

0 

.1711 

0  . 

0791 

2 

.  1622 

0 

.0318 

AvgAi rc  raftHrsInSvc 

0 

.0000 

0  . 

0000 

1 

.6090 

0 

.1092 

Loc 

0 

.2089 

0  . 

0364 

5 

.  7472 

0 

.0000 

TimelnSqdnMed 

-0 

.0203 

0  . 

0043 

-4 

.  7092 

0 

.0000 

DeploymentUDP 

-0 

.0697 

0  . 

0429 

-1 

.6253 

0 

.1057 

DeploymentCVN 

0 

.1934 

0  . 

1025 

1 

.  8868 

0 

.0606 

Deployment IRAQ 

0 

.1214 

0  . 

1568 

0 

.7741 

0 

.4398 

Residual  standard  er 

ro: 

r:  0  .: 

2105 

on  201 

deg  ree  s 

of 

freed 

Multiple  R-Squared:  0.2642 

F-statistic:  9.021  on  8  and  201  degrees  of  freedom,  the  p-value  is  1.464e-010 


Analysis  of  Variance  Table 


Response:  log (MMHperMA) 


Terms  added  sequentially 

Df 

TEC  2 

AvgAircraftHrsInSvc  1 
Loc  1 
TimelnSqdnMed  1 
Deployment  3 
Residuals  201 


(first  to  last) 

Sum  of  Sq  Mean  Sq  F  Value 
0.286042  0.143021  3.22892 

0.137936  0.137936  3.11411 

1.365617  1.365617  30.83090 
1.098606  1.098606  24.80272 
0.308534  0.102845  2.32188 

8.903050  0.044294 


Pr  (F ) 
0 .04166326 
0 .07913569 
0 .00000009 
0 .00000136 
0 .07634081 


Figure  23  Stepwise  Variable  Selection,  S-Plus  Report. 


We  conduct  residual  analysis  on  this  reduced  model  to  check  the 

assumptions  of  linear  regression.  Appendix  F  shows  three  residual  plots  for  this 
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regression:  the  residuals  against  the  fitted  values,  the  responses  against  the 
fitted  values,  and  the  normal  quantile-quantile  (QQ)  plot  of  the  residuals.  As  seen 
by  these  plots,  the  residuals  appear  to  have  fairly  constant  variance  and  normal 
distribution,  their  variance  appears  to  be  constant,  and  they  appear  to  be 
independent  of  the  fitted  values. 

3.  Unexplained  Variability  in  the  Performance  Measure 

As  noted  above,  this  model  explains  only  18  percent  of  the  variability  of 
the  man-hours  per  maintenance  action  performance  measure.  In  its  current  form, 
we  have  left  the  organization  term  out  of  the  model,  suggesting  that  the  model 
applies  to  all  squadrons.  We  check  the  distribution  of  the  residuals  by  squadron 
to  know  whether  this  is  a  valid  conclusion.  This  plot  is  shown  in  Figure  24  . 


0.5 


0.0 


-0.5 


VMFA115  VMFA122  VMFA225  VMFA242  VMFA312  VMFA323  VMFA533 

VMFA121  VMFA224  VMFA232  VMFA251  VMFA314  VMFA332 

Figure  24  Boxplots  of  Residuals  Grouped  by  Squadron,  Stepwise  Reduction 
Model. 

The  box  represents  all  data  between  the  25th  and  75th  percentiles.  The  line  inside  the  box 
represents  the  median  of  the  distribution.  The  vertical  lines  that  extend  above  and  below 
the  box  represent  the  range  of  data;  dots  outside  these  ranges  represent  outliers. 

The  plot  indicates  that  the  residuals  are  not  evenly  distributed  among  the 
squadrons.  The  model  tends  to  under-predict  MMHperMA  for  VMFA232,  for 
example,  and  over-predict  for  VMFA323.  We  conclude,  therefore,  that  there  is 
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variability  in  the  residuals  that  is  attributed  to  squadron  characteristics  that  we 
have  not  accounted  for  with  this  model. 

We  measure  the  additional  information  carried  by  the  organization  term  by 
including  the  term  and  fitting  a  new  model.  The  ANOVA  table  from  the  output  is 
shown  in  Figure  25  . 


***  Linear  Model  *** 


Call:  lm( formula  =  log (MMHperMA)  -  AvgAircraftHrsInSvc  +  TimelnSqdnMed  + 


Deployment  +  Org, 

data  = 

A,  na . 

action  =  na 

. excludi 

Residuals : 

Min  IQ 

Median 

3Q 

Max 

-0.7842  -0.1022 

-0.0007759  0.1025 

0.5435 

Coefficients : 

Value  Std 

.  Error  t  value  Pr 

( Intercept ) 

1.0106 

0 . 2628 

3.8456 

0.0002 

AvgAircraftHrsInSvc 

0.0000 

0 . 0000 

0.7351 

0.4631 

TimelnSqdnMed 

-0.0067 

0 . 0034 

-1.9337 

0.0544 

DeploymentUDP 

-0.0585 

0 . 0353 

-1.6577 

0.0988 

Deployment  CVN 

0.1494 

0 . 0532 

2.8079 

0.0054 

Deployment I RAQ 

-0.1167 

0 . 0866 

-1.3475 

0.1792 

OrgVMFA122 

-0.1439 

0 .0755 

-1.9054 

0.0580 

OrgVMFA224 

0.1033 

0.0966 

1.0688 

0.2863 

OrgVMFA22  5 

0.1367 

0 . 1366 

1.0007 

0.3181 

OrgVMFA232 

0.2641 

0 . 0873 

3.0261 

0.0028 

OrgVMFA24  2 

0.1941 

0 . 1057 

1.8366 

0.0676 

OrgVMFA25 1 

-0.2530 

0 . 0973 

-2.5988 

0.0100 

OrgVMFA312 

-0.2043 

0 . 0589 

-3.4655 

0.0006 

OrgVMFA314 

0.1929 

0 . 1307 

1.4761 

0.1413 

OrgVMFA323 

-0.1907 

0 . 0899 

Mpi200 

0.0351 

OrgVMFA332 

0.1114 

0 . 1144 

0.9737 

0.3312 

OrgVMFA533 

-0.1063 

0 . 1010 

S|p0524 

0.2938 

Residual  standard  error:  0.1835  on  223  degrees  of  freedom 
Multiple  R-Squared:  0.4887 

F-statistic:  13.32  on  16  and  223  degrees  of  freedom,  the  p-value  is  0 
163  observations  deleted  due  to  missing  values 


Figure  25  Reduced  Model,  Organization  Term  Included,  S-Plus  Report 


The  R2  value  has  increased  to  0.4887,  which  is  double  the  explanatory 
power  than  the  model  without  the  organization  term.  We  use  ANOVA  to  test  the 
significance  of  the  added  term.  The  results  of  the  ANOVA  test,  shown  in  Figure 
26  ,  indicate  that  the  model  with  the  organization  term  is  significantly  better  at 
explaining  variability  of  man-hours  per  maintenance  action  than  the  model 
without  the  organization  term. 
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Note  that  TEC  and  Location  have  been  removed  from  the  model  before 
inclusion  of  the  organization  term;  this  is  because  these  variables  are  uniquely 
determined  by  the  organization  variable.  If  we  include  all  three  terms,  we  face 
singularity  in  the  design  matrix  used  to  calculate  the  least  squares  solution. 


>  anova(lm3,lm3plusOrg) 

Analysis  of  Variance  Table 

Response:  In(MMHperMA) 

Terms 

1  TEC  +  Avg  Ai  rcraftH  rs  1  nS  vc  +  Loc  +  TimelnSqdnMed  +  Deployment 

Resid.  Df 
231 

RSS  Test  Df 

11,21171 

Sum  of  Sq  F  Value 

Pr(F) 

2  AvgAircraftHrsInSvc  +  TimelnSqdnMed  +  Deployment  +  Org 

223 

7.50948  1  vs.  2  8 

3.702232  13.74259 

3.330669e-016 

Figure  26  ANOVA  Test  for  Significance  of  Added  Organization  Term. 
4.  Measuring  Tech  Rep  Effects  with  ELAR 


We  want  to  know  if  the  volume  of  tech  rep  activity  affects  the  performance 
of  the  squadrons.  We  use  the  number  of  ELAR  records  in  a  given  month  as  a 
measure  of  tech  rep  activity.  As  observed  in  Section  C  of  this  chapter,  users  of 
ELAR  have  been  regularly  recording  the  squadron  field  only  recently,  giving  us 
only  several  months  of  records  to  which  we  can  attribute  to  a  specific  squadron. 
The  method  used  above  is  applied  to  test  if  there  is  additional  explanatory  power 
when  adding  the  ELAR  term  to  the  model.  We  test 


H o :  In  ys>,  =  fi0  +  +  P2X 2>I>,  +  P3X3  s  t  +  /34X^t  +  P5X5^  +  frX^  +  (4) 


against  the  alternative 


Ha :  In  Ys  t  =  fi0  +  (5xXXst  +  P2X2  s  t  +  p,X3s  l  +  P4X4  s  t  +  f35X5  s  l  + 

P6X(,,s,t  +  +  £s,t 


where 


(5) 


69 


7  t  =  man-hours  per  maintenance  action,  squadron  s,  month  t 
X\st=  type  equipment  code 
X2st  =  upper  quartile,  months  experience 
X3sr  =  location 

X4st  =  median,  months  in  squadron 
X5sl  =  deployment  status 
X6  s  t  =  flight  hours 
X7  s  t  =  ELAR  records 
es ,  =  residual 


We  then  use  ANOVA  to  test  for  the  difference  between  the  two  models.  We 
further  test  for  the  individual  affects  of  the  various  types  of  tech  rep  activities  by 
counting  ELAR  records  by  problem  type  and  giving  each  problem  type  its  own 
term  in  the  model.  In  this  way  we  differentiate  between  the  effects  of  training- 
related  activities  that  precede  aircraft  malfunctions  and  those  repair-related 
activities  that  follow  aircraft  malfunctions.  With  several  years’  worth  of  additional 
observations,  entered  regularly  by  all  squadrons,  we  believe  that  the  techniques 
employed  here  will  be  effective  in  identifying  those  areas  where  tech  rep  activity 
has  improved  maintenance  performance. 

5.  Lag  Effects 

The  various  squadron  characteristics  that  explain  squadron  performance 
might  not  immediately  affect  the  performance  metric.  If  we  believe  that  some  of 
the  predictors  exhibit  a  delayed  effect  on  performance,  then  we  lag  those 
individual  variables  backward  by  the  appropriate  time  interval.  For  our  purposes 
we  lag  the  response  variable  man-hours  per  maintenance  action  forward  by  one 
month,  to  check  whether  the  explanatory  variables  have,  in  general,  have  a  one- 
month  delay  until  reflected  in  the  performance  measure.  The  resulting  model, 
formally,  is 


ln  Ys,t+ 1  -  A  +  P\X\ ,s,t  +  PlX2 ,s,t  +•••  +  PkXk,s,t  +  £s,t 


(6) 


which  is  equivalent  to  lagging  the  predictors  backward  a  month: 

,n  Ys,t  =  A)  +  P\X\,s,t-\  +  PlX2,s,t-\  +-+  PkXk,s,t-\  +£s,t 
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(7) 


We  are  using  the  performance  metric  man-hours  per  maintenance  action 
specifically  because  we  believe  that  it  is  a  leading,  not  a  lagging,  indicator.  That 
is,  we  expect  little  delay  between  changes  in  squadron  characteristics  and  the 
resultant  change  in  the  leading  indicator.  For  this  reason,  we  do  not  pursue  the 
issue  of  lagging  variables  further  in  this  analysis. 

6.  Time  Effects 

Although  we  normally  consider  the  possibility  of  a  time-of-year  effect  in  a 
response  variable  as  observed  over  the  course  of  time,  the  performance  variable 
in  this  case,  man-hours  per  maintenance  action  should  not  be  affected  by  the 
month  or  quarter  during  which  it  is  measured.  We  can  think  of  no  reason  why  the 
squadron  performance  would  fluctuate  by  quarter.  The  environmental  changes 
experienced  by  Marine  squadrons  is  affected  more  by  the  geographic  location 
associated  with  their  current  operating  base,  whether  at  home  base  or  deployed, 
than  it  is  by  the  season.  Furthermore,  to  detect  monthly  or  quarterly  effects,  we 
need  several  years’  worth  of  data.  For  these  reasons,  we  do  not  include  a  month 
or  quarter  factor  in  the  model. 

The  data  for  this  analysis  were  collected  as  time  series  observations  for 
each  of  the  squadrons  under  investigation.  Each  record,  therefore,  has  an 
associated  month,  quarter,  and  year.  For  time  series  data,  we  check  for  serial 
correlation  of  the  residuals  to  ensure  that  we  do  not  have  time  patterns  in  the 
residuals,  indicating  a  time  effect  of  some  sort.  Presence  of  serial  correlation  may 
cast  doubt  on  the  reliability  of  estimates  derived  from  the  fitted  model.  The 
Durbin-Watson  test  is  a  common  test  for  detecting  serial  correlation  of  the 
residuals  resulting  from  regression  models.  Draper  and  Smith  [1981]  provide  a 
description  of  this  test.  We  usually  assume  that  the  residuals  from  a  linear  model 
are  independent  and  normally  distributed,  and  that  all  serial  correlations,  ps=  0. 

We  test  the  null  hypothesis  H0:ps  =0  against  the  alternative,  Ha:ps=p\  We 
use  the  Durbin-Watson  statistic, 
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dw  = 


Z(e,  -«’,i  f 

±<e,-ef 


to  determine  whether  our  residuals  call  for  us  to  reject  the  null  hypothesis.  Since 
our  data  columns  are  observations  which  have  been  stacked  by  squadron,  we 
“unstack”  the  data  and  treat  each  squadron  separately  as  its  own  time  series  of 
observations  with  its  own  set  of  residuals.  For  each  of  these  sets  of  residuals  we 
form  the  Durbin-Watson  statistic  dw ,  and  reject  H0  if  dw  is  below  a  critical  value 

obtained  from  tables  such  as  those  published  in  Draper  and  Smith  [1981],  Use  of 
these  tables  requires  three  parameters:  the  level  of  significance,  the  number  of 
variables  and  the  number  of  observations.  In  our  case,  for  a  =  0.05,  n  =20,  and 
k  =  4,  we  obtain  a  critical  value  of  dw  =  1.70.  For  a  two-sided  test  against 
alternatives  p  a  0 ,  if  dw  <  dwcrir  or  if  4  -  dw  <  dwcnt ,  we  reject  H0  at  level  2a.  We 
calculate  the  statistic  for  each  of  the  squadrons,  and  the  results  are  as  follows: 


dw 

1-dw 

n 

[1,  ] 

1 

.  679851 

2 

.320149 

20 

[2,  ] 

1 

.  618424 

2 

.381576 

20 

[3,  ] 

1 

.  558493 

2 

.441507 

20 

[4,  ] 

2 

.  140955 

1 

.859045 

20 

[5,  ] 

1 

.318383 

2 

.  681617 

20 

[6,  ] 

1 

.  535749 

2 

.464251 

20 

[7,  ] 

1 

.  636940 

2 

.363060 

20 

[8,  ] 

2 

.  159661 

1 

.840339 

20 

[9,  ] 

1 

.  933646 

2 

.066354 

20 

[10,  ] 

1 

.388049 

2 

.  611951 

20 

[11,  ] 

1 

.316829 

2 

.  683171 

20 

[12,  ] 

2 

.  153625 

1 

.846375 

20 

Figure  27  Durbin-Watson  Test  of  the  Residuals 

dw:  the  vector  of  12  Durbin-Watson  statistics  calculated  from  the  residuals 
of  each  squadron’s  observations.  VMFA-121  has  been  omitted, 
n:  the  number  of  observations  for  each  squadron. 

4-dw:  used  to  test  the  lower  side  of  the  2-sided  Durbin-Watson  test. 


72 


For  a= 0.025,  we  expect  approximately  5%  of  our  2-sided  tests  to  reject  H0.  In 

our  case,  we  see  that  eight  of  the  twelve  squadrons  have  resulted  in  rejection  of 
the  null,  which  leads  us  to  believe  that  we  have  some  presence  of  autocorrelation 
of  the  residuals. 
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IV.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

The  primary  objective  of  this  thesis  is  to  identify  F/A-18  squadron 
characteristics  that  are  important  predictors  of  maintenance  performance.  A 
secondary  objective  is  to  draw  insights  on  performance  from  data  collected  by 
NATEC,  in  ELAR,  on  the  utilization  of  engineering  and  technical  services.  The 
two-year  time  frame  of  our  study  was  a  limiting  factor  in  discerning  relationships 
for  two  reasons:  it  necessarily  restricted  the  levels  of  change  that  were  possible 
in  the  squadron  attributes  that  were  measured;  and  in  the  case  of  ELAR,  it 
implied  that  the  available  data  represented  the  “learning  curve”  of  the  system. 
Nonetheless,  our  analysis  should  provide  a  useful  template  for  future  studies  with 
longer  time  series  and  with  data  of  higher  quality. 

In  the  following  subsections  we  address  the  five  research  questions  that 
we  posed  in  Chapter  1 . 

1.  Significant  Variables 

Which  squadron  characteristics  have  a  detectable  contribution  to  the 
variability  of  the  performance  measure  man-hours  per  maintenance  action ? 

From  those  variables  included  in  the  model  selection  process,  five  are  found  to 
be  statistically  significant  in  explaining  at  least  some  of  the  variability  of  the 
performance  metric  of  this  study,  man-hours  per  maintenance  action'. 

•  Type  Equipment  Code 

•  Average  aircraft  hours  in  service 

•  Location 

•  Median,  months  in  squadron 

•  Deployment  status 

The  linear  model  including  these  variables  explains  approximately  28 
percent  of  the  variability  of  the  natural  logarithm  of  man-hours  per  maintenance 
action.  We  used  a  logarithm  transformation  to  better  meet  the  assumptions  of  a 
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normal,  linear  model.  For  this  study,  only  20  monthly  observations  for  each  of  the 
thirteen  U.S. -based  active  duty  Marine  Corps  F/A-18  squadrons  were  complete 
with  no  missing  values. 

2.  Squadron  Differences 

How  much  additional  variability  is  explained  by  the  squadron  that  is 
not  accounted  for  by  the  squadron  characteristics  already  considered?  To 

answer  this  question,  we  tested  for  a  significant  difference  between  two  models: 
one  without  the  organization  term,  and  the  same  model  with  an  organization  term 
added.  We  find  that  by  including  organization  in  the  model,  we  are  defining  a 
different  fit  for  each  squadron.  We  gain  significant  additional  predictive  power 
with  the  inclusion  of  this  term.  The  value  of  R2  is  improved  from  approximately 
0.24  to  0.48,  which  tells  us  that  the  squadrons  are  different  in  ways  for  which  our 
variables  do  not  account.  There  is  important  information  in  both  models.  Without 
the  organization  term,  we  have  a  model  that  applies  to  all  squadrons.  This  model 
would  therefore,  presumably,  apply  to  any  squadron  if  its  characteristics  were 
similar  in  general  to  those  that  formed  the  model.  If  we  instead  allow  for  a 
different  fit  for  each  squadron,  by  adding  the  organization  term,  we  obtain  a 
model  that  can  be  used  to  predict  changes  in  man-hours  per  maintenance  action 
as  conditions  change  within  a  particular  squadron. 

3.  Time  Effects  and  Autocorrelation 

Is  there  a  time-of-year  effect  for  the  performance  of  the  squadrons? 

We  do  not  find  the  quarter  term  to  be  significant  in  the  model,  so  we  conclude 
that  all  quarters  are  essentially  the  same.  However,  this  study  is  limited  to  20 
months  of  complete  observations,  which  is  a  relatively  small  set  of  data  to  test  for 
a  quarterly  effect.  Through  employment  of  the  Durbin-Watson  test,  we  do  detect 
a  slight  correlation  of  the  residuals,  suggesting  that  the  residuals  are  not 
independent.  There  is  a  temporal  structure,  although  slight,  which  could  be 
handled  with  a  generalized  least  squares  approach. 
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B.  RECOMMENDATIONS 

1.  Additional  Variables 

What  additional  metrics  not  currently  available  would  most  likely  be  useful 
in  an  explanatory  model  of  maintenance  performance?  Our  methods  depend  on 
the  aggregation  of  data  by  month  and  on  the  use  of  aviation  maintenance  metrics 
currently  available  from  NALCOMIS.  However,  many  metrics  could  be  derived 
with  direct  access  to  the  actual  records  in  NALCOMIS.  Since  we  are  trying  to 
measure  maintainer  capability,  we  would  like  to  have  as  many  metrics  that 
quantify  this  capability  as  possible.  Several  of  the  metrics  currently  unavailable 
that  would  likely  be  useful  are  repeat  discrepancy  rate,  the  fix  rate,  and  the 
maintenance  efficiency  rate.  These  metrics  are  recognized  by  Air  Force 
maintenance  analysts  for  their  importance  [AFLMA,  2001],  The  repeat 
discrepancy  rate  gives  an  indication  of  those  malfunctions  that  were  thought  to 
have  been  repaired  but  were  not,  in  effect,  repaired  correctly.  The  fix  rate  is  the 
ratio  of  critical  discrepancies  repaired  to  the  total  critical  discrepancies  received, 
where  critical  discrepancies  are  those  that  place  the  aircraft  in  a  not-flyable 
status.  The  maintenance  efficiency  ratio  is  the  maintenance  actions  completed  as 
a  percentage  of  those  scheduled  (in  a  given  time  period). 

2.  ETS  Data  Collection 

What  data  collection  methods,  if  any,  would  be  likely  to  improve  the  ability 
of  NATEC  managers  to  correlate  squadron  characteristics  to  tech  rep  measures 
of  performance?  Our  goal  is  to  identify  tech  rep  activity  that  correlates  with 
measures  of  squadron  performance.  To  achieve  this  goal,  we  need  to  link  tech 
rep  activity  to  the  maintenance  activity  that  the  tech  reps  are  assisting  and  to  the 
performance  measure  that  we  are  analyzing.  To  link  the  tech  rep  activity  to  the 
specific  maintenance  activity  under  investigation,  we  need  a  reliable  (preferably 
automated)  means  of  identifying  specific  repair  actions  with  a  tech  rep.  To  that 
end,  we  need  one-to-one  relationships  between  the  records  in  ELAR,  or  its 
equivalent,  and  those  in  NALCOMIS. 

This  can  be  achieved  in  several  ways.  The  (ELAR)  database  could  require 
that  an  accurate  job  control  number  (JCN)  from  the  corresponding  NALCOMIS 
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record  be  entered  into  each  ELAR  record.  The  JCN  is  a  unique  number  that 
would  clearly  identify  the  maintenance  action  to  which  the  ELAR  record  applies. 
A  more  effective  solution  would  be  for  NALCOMIS  to  integrate  any  tech  rep 
activity  directly  with  maintenance  action.  For  instance,  there  is  a  non-mandatory 
(and  therefore  seldom-used)  block  on  the  maintenance  action  form  (MAF)  by 
which  tech  rep  assistance  can  be  identified.  More  effective,  perhaps,  would  be  to 
expand  the  capability  of  the  MAF  to  allow  for  additional  tech  rep  details,  such  as 
the  name  of  the  tech  rep,  the  type  of  assistance  rendered,  the  actual  start  time 
and  end  time  of  the  tech  rep  action,  and  any  other  customer  service-type 
information  that  could  quickly  be  added  at  the  data  point-of-entry.  Another 
alternative  would  be  to  require  that  the  tech  rep  document  his  or  her  actions  in 
NALCOMIS  before  the  MAF  can  be  approved  by  maintenance  control 
supervisors. 

3.  Real-Time  Maintenance  Proficiency 

A  large  part  of  the  tech  reps’  value  to  the  squadrons  is  in  their  training 
role.  Tech  reps  fill  the  gaps  between  initial  MOS  training  and  formal  follow-on 
school  training  that  is  not  available  to  every  maintainer.  For  that  reason,  there  is 
a  need  to  more  accurately  quantify  the  training  level  of  the  squadron  in  general 
and  of  each  maintainer  specifically.  In  other  words,  at  any  given  time,  we  need  to 
be  able  to  obtain  a  picture  of  the  training  levels  across  a  squadron’s  maintenance 
department.  For  aircrews,  this  is  achieved  through  the  use  of  a  training  and 
readiness  (T&R)  syllabus  and  through  the  correct  demonstration,  at  regular 
intervals,  of  mission  essential  tasks.  The  maintainers  need  an  analogous  list  of 
mission  essential  tasks,  specific  to  their  MOS’s,  which  need  to  be  performed  at 
prescribed  intervals  to  maintain  proficiency  and  “currency.”  Each  repair  they 
perform,  either  on  actual  discrepancies  or  in  a  training  setting,  “updates”  the 
currency  of  qualification  in  that  specific  area  of  repair.  At  any  point  in  time,  the 
commander  (or  NATEC)  could  see  areas  of  maintenance  training  that  have  not 
been  accomplished  in  some  time  (approaching  expiration)  and  must  therefore  be 
addressed  through  training.  In  this  way,  limited  tech  rep  resources  could  be 
dedicated  to  preemptive  training  in  those  skill  areas  deemed  to  be  critical  and 
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fleeting,  rather  than  always  reacting  to  an  actual  aircraft  malfunction  that 
demands  unscheduled  maintenance. 

C.  OPPORTUNITIES  FOR  FURTHER  STUDY 

1.  Analysis  of  NALCOMIS  Records 

Analysis  of  several  years’  of  NALCOMIS  data  records,  which  would 
include  millions  of  individual  flight  and  maintenance  records,  would  call  for 
analytical  techniques  not  addressed  here.  Algorithmic  statistical  methods,  such 
as  clustering,  classification  trees,  and  neural  networks,  could  be  employed  to  find 
patterns  in  the  data,  which  might  demonstrate  better  predictive  performance  than 
traditional  regression  methods.  For  instance,  we  could  use  these  techniques  to 
predict  whether  or  not  a  maintenance  action  of  a  certain  type  will  require 
technical  assistance.  Other  large  data  sets,  such  as  supply  records,  could  be 
incorporated  with  similar  techniques.  For  any  of  these  techniques  to  be  useful 
with  respect  to  tech  rep  performance,  for  the  reasons  discussed  in  Section  B.3  of 
this  chapter,  the  records  need  to  have  some  direct  link  to  tech  rep  activity. 

2.  Survey  of  Tech  Rep  Customers 

Current  indicators  of  customer  satisfaction  are  those  comments  obtained 
from  users  of  tech  rep  services  at  the  completion  of  a  tech  rep  action.  For 
reasons  discussed  in  Chapter  I,  the  users  of  tech  rep  assistance  have  every 
incentive  to  request  continued  ETS  support,  since  they  do  not  “pay”  for  that 
support.  Customers  are  understandably  reluctant,  therefore,  to  submit  comments 
that  will  jeopardize  the  continued  availability  of  tech  reps.  If,  on  the  other  hand, 
the  customers  (squadrons)  are  forced  to  make  tradeoff  decisions  regarding 
resources,  we  might  see  a  different  picture.  One  way  to  obtain  a  more  objective 
input  would  be  to  provide  squadron  commanders  a  fictitious  “budget”  that  can  be 
allocated  towards  those  resources  available  to  them  but  that  they  do  not  normally 
have  to  pay  for:  additional  fuel,  flight  hours,  personnel,  repair  parts  and 
consumables,  and  technical  services.  In  this  light  we  could  learn  the  true 
importance  of  technical  services  to  those  that  have  many  other  requirements  as 
well. 
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APPENDIX  A 

MISSION  ESSENTIAL  SUBSYSTEMS  MATRIX,  F/A-18 


OPNAVINST  5442.4M 

1 7  OCT  1990 

F/A-18A/B/C/D 

TYPE  EQUIPMENT  CODES:  AMAA/AMAE/AMAF/AMAG 

Do  not  assign  an  EOC  code  if  all  equipment  is  operational.  The 
aircraft  is  OPC. 

Assign  alpha  character  (B)  of  the  EOC  code  when  the  following 
system (s)  are  inoperative.  The  aircraft  is  FMC,  M  or  S. 

AUTOMATIC  DIRECTION  FINDER  SET 
CLOCK 

EXTERNAL  POWER  SYSTEM 
MAGNETIC  COMPASS  (AQU-3/A) 

STRIKE  CAMERA  SYSTEM  (NOTE  1) 

VIDEO  TAPE  RECORDER  (NOTE  1) 

Assign  alpha  character  (C)  of  the  EOC  code  when  the  following 
system(s)  are  inoperative  preventing  the  escort/ strike  mission. 
The  aircraft  is  not  capable  of  independent  detection  and 
destruction  of  aircraft/missiles  under  all-weather  conditions 
or  providing  protective  escort  for  strike  and  support  forces 
using  all  air-to-air  weapons  in  a  multi-threat  ECM  environment. 
The  aircraft  is  PMC,  M  or  S. 

LAU-116 
LAU-117 

MISSILE  ILLUMINATION  GROUP  SPARROW 
SPARROW  MISSILE  EJECTOR  LAUNCHER 

Assign  alpha  character  (D)  of  the  EOC  code  when  the  following 
system(s)  are  inoperative  preventing  the  strike  mission.  The 
aircraft  is  not  capable  of  conducting  interdiction  or  war-at- 
sea  missions  using  all  weapons  and  delivery  modes  compatible 
with  aircraft  regardless  of  terrain,  weather  or  enemy  defenses. 
The  aircraft  is  PMC,  M  or  S. 

AMAC  SYSTEM  (NOTES  1,2) 

WEAPON  CONTROL  (HUD,  MULTIPURPOSE  DISPLAY 

GROUP)  (NOTE  3) 

Assign  alpha  character  (J)  of  the  EOC  code  when  the  following 
system (s)  are  inoperative  preventing  the  visual  attack  mission. 
The  aircraft  is  not  capable  of  conducting  missions  under  VMC, 
using  system  deliveries  of  conventional  ordnance,  conducting 
anti-radiation  missile  strike  support,  close  air  support  for 
friendly  forces  with  forward  air  controller.  The  aircraft  is 
PMC,  M  or  S. 


(NOTE  1) 
(NOTE  1) 


Enclosure  (1)  214 

Figure  A-1.  Chief  of  Naval  Operations  Instruction  5442.4M  (OPNAVINST 
5442. 4M).  (1992).  Aircraft  material  condition  definitions,  mission-essential 
subsystems  matrices  (MESMS),  and  Mission  Descriptions.  01  July  1992. 
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OPNAVXNST  5442 

17  OCT  1990 

F/A-18A/B/C/D/  (cont) 


ACM  (AWG-25) (HARM)  (NOTE 

ANTX-G  SUIT  PRESSURE 
ARMAMENT  CONTROL  PROC  SET 

CENTERLINE  PYLON  (SUU-62)  (NOTE 

CHAFF  COUNTERMEASURES  SET  (ALE-39)  (NOTE 

COUNTERMEASURES  SET  (AWJ-126  OR  ALQ-165)  (NOTE 

DATA  STORAGE  UNIT 
DIGITAL  MAP 
EW  THREAT  DISPLAY 

FLIR  POD  AND  ADAPTER  (NOTE 

HAVEQUI CK/ S INCGARS  (ARC-182) 

HORIZONTAL  INDICATOR  (NOTE 

LASER  SPOT  TRACKER  AND  ADAPTER  (NOTE 

LASER  TARGET  DESIGNATOR/RANGER  (NOTE 

NAVIGATION  FLIR 
NIGHT  VISION  GOGGLES 
PALLETIZED  GUN  SYSTEM  (M61A1) 

RADAR  LIQUID  COOLING  SYSTEM 

RADAR  SET  (APG-65)  (NOTE 

RADAR  WARNING  RECEIVER  (ALR-67)  (NOTE 

SELECTIVE  STORES  JETTISON  SYSTEM 
SIDEWINDER  MISSILE  SYSTEM 

SIDEWINDER  LAUNCHERS  (LAU-7A)  (NOTE 

THREAT  WARNING  LIGHT  DISPLAY  GROUP 

WEAPON  RELEASE  RACKS  (BRU-32/A  or  BRU-32A/A)  (NOTE 

WEAPON  SYSTEM  CONTROL  FUNCTION  (HOTAS)  (NOTE 

WING  PYLONS  (SUU-63)  (NOTE 


.  4M 


1) 


1) 

1) 

1) 


1) 

4) 

1) 

1) 


4) 

1) 


1) 

1) 

4) 

1) 


Assign  alpha  character  (K)  of  the  EOC  code  when  the  following 
system (s)  are  inoperative  preventing  the  expanded  mobility 
mission.  The  aircraft  is  not  capable  of  safe  movement  on  and 
off  CV/SATs  during  day,  night  and  inclement  weather  conditions, 
conducting  independent  navigation,  using  encrypted  radio  voice 
communications  and  IFF,  or  in-flight  refueling  (receive) .  The 
aircraft  is  PMC,  M  or  S. 


AIR  REFUELING  PROBE 

AIR  REFUELING  PROBE  FLOOD  LIGHT 

ANGLE-OF-ATTACK  SYSTEM  AND  INDEX  LIGHTS 

APPROACH  POWER  COMPENSATOR  SYSTEM 

BOARDING  LADDER  DRAG  BRACE 

CATAPULT  SYSTEM 

EMERGENCY  JETTISON  SYSTEM 

ILS  RECEIVER/DECODER  (ARA-63)  (NOTE  5) 


2X5  Enclosure  (1) 

Figure  A-1.  Chief  of  Naval  Operations  Instruction  5442.4M  (OPNAVINST 
5442. 4M).  (1992).  Aircraft  material  condition  definitions,  mission-essential 
subsystems  matrices  (MESMS),  and  Mission  Descriptions.  01  July  1992. 
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OPNAVINST  5442.4M 


17  OCT  1990 


F/A— 18  A/B/C/D  (COOT) 


MISSION  COMPUTERS  (BOTH  REQUIRED) 

RADAR  BEACON  (APN-202)  AND  RT  (102B/APN-202) 
RECEIVER  TRANSMITTER  PROCESSOR  (RT-1379/ASW) 
SECONDARY  POWER  SUPPLY  (APU) 

SECURE  IFF  (KIT  1A) (MODE  4) 

SECURE  VOICE  (KY-58) 

WING  FOLD 


(NOTE  5) 
(NOTE  5) 

(NOTE  1) 
(NOTE  1) 


Assign  alpha  character  (L)  of  the  EOC  code  when  the  following 
system (s)  are  inoperative  preventing  the  IMC  flight  mission. 

The  aircraft  is  not  capable  of  day  or  night  IMC  field  flight 
operations  with  necessary  communications,  IFF,  navigation, 
flight  and  safety  systems  required  by  applicable  NATOPS  and  FAA 
regulations.  The  aircraft  is  PMC,  M  or  S. 


ALTIMETER,  ELECTRONIC  (APN-194(V)) 

ENGINE  ANTI-ICE  SYSTEM 

EXTERIOR  LIGHTING  (POSITION  AND  FORMATION)  (NOTE  6) 

IFF  TRANSPONDER  (APX-IOO(V)) 

INTERIOR  LIGHTING 

MISSION  COMPUTERS  (AYK-14) (MCI  REQUIRED) 
PITOT/ANGLE-OF-ATTACK  PROBE  HEATER  SYSTEM 
TACTICAL  NAVIGATION  SET  (ARN-118(V)) 

TAXI  LIGHT 
UP  FRONT  CONTROL 

WINDSHIELD  ANTI-ICE  AND  RAIN  REMOVAL 
WHEEL  ANTI-SKID  CONTROL  SYSTEM 


Assign  alpha  character  (Z)  of  the  EOC  code  when  the  following 
system ( s ) /condition ( s )  prevent  the  aircraft  from  being  safely 
flyable.  The  aircraft  is  not  capable  of  day,  field  flight 
operations  under  VMC  with  two-way  radio  communication  and 
necessary  aircraft  and  crew  safety  provisions.  The  aircraft  is 
NMC,  M  or  S. 


AIR  CONDITIONING/PRESSURIZATION 
AIRFRAME 

BOMBING  NAVIGATION  (INS) 

CSC 

DECELERATION  EQUIPMEOT/DROGUE  PARACHUTE 
ELECTRICAL  SYSTEMS 
EMERGENCY/PARKING  BRAKE 
EMERGENCY  EQUIPMENT 
EMERGENCY  RADIO 
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Figure  A-1.  Chief  of  Naval  Operations  Instruction  5442.4M  (OPNAVINST 
5442. 4M).  (1992).  Aircraft  material  condition  definitions,  mission-essential 
subsystems  matrices  (MESMS),  and  Mission  Descriptions.  01  July  1992. 
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F/A-18A/B/C/D  (cont) 


OPNAVINST  5442. 4M 

17  OCT  1990 


EMI  PROTECTION  DEVICES 
ENGINES 

EXPLOSIVE  DEVICES 

FLIGHT  CONTROLS 

FLIGHT  REFERENCE 

FUEL  SYSTEM  (FUSELAGE  AND  WINGS) 

FUSELAGE  COMPARTMENTS 
HYDRAULIC/PNEUMATIC  SYSTEM 
ICS  (REQUIRED  IN  F/A-I8B  AND  F/A-18D) 
INSTRUMENTS/INSTRUMENT  SYSTEM  (WUC  51  SERIES) 
INTEGRATED  GUIDANCE  AND  FLIGHT  CONTROL 
(ASN-130  OR  ASN-139) 

LANDING  GEAR 

LIGHTING  SYSTEMS  (ANTI-COLLISION  LIGHT) (2  MINIMUM) 

MAINTENANCE  SIGNAL  DATA  RECORDER  SET 

MISCELLANEOUS  UTILITIES 

OXYGEN  SYSTEMS/OBOGS 

POWER  PLANT  INSTALLATION 

STRAIN  GAUGES 

UHF  COMMUNICATION  SYSTEMS  (1  REQUIRED) 

WEAPON  CONTROL  (HEAD-UP  DISPLAY,  MULTIPURPOSE 
DISPLAY  GROUP) 

WEAPON  DELIVERY 
CONDITIONAL  INSPECTION 
ENGINE  INSPECTION 
PHASE  INSPECTION 
SPECIAL  INSPECTION 
TECHNICAL  DIRECTIVE  COMPLIANCE 


(NOTE  7) 


(NOTE  8) 


(NOTE  9) 


(NOTE  10) 


(NOTE  3) 

(NOTE  11) 
(NOTE  11) 
(NOTE  11) 
(NOTE-11) 
(NOTE  11) 


NOTES : 


1. 


2. 

3. 

4. 

5. 


6. 


WHEN  THE  EQUIPMENT  IS  INSTALLED,  REPORT  ON  THE  COMPLETE 
SYSTEM.  WHEN  THE  EQUIPMENT  IS  NOT  INSTALLED,  REPORT  ON  THE 
WIRING  AND  PLUMBING  ONLY. 

NOT  APPLICABLE  TO  VMFA  SQUADRONS  UNLESS  ASSIGNED  TO  CVW. 
HUD,  LEFT  AND  RIGHT  DDI,  KI  REQUIRED  FOR  MISSIONS  A,  B,  C 
AND  D.  HUD  AND  LEFT  DDI  REQUIRED  FOR  MISSIONS  J,  K  AND  L. 
ALL  AIR-TO-AIR,  ACM,  AND  AIR-TO-GROUND  MODES  REQUIRED. 
EITHER:  RADAR  BEACON  (APN-202  AND  RT  (1028/APN-202) ,  OR  ILS 
RECEIVER/DECODER  (ARA-63)  AND  RECEIVER  TRANSMITTER 
PROCESSOR  (RT— 1379/ASW)  REQUIRED. 

ONLY  REQUIRED  TO  BE  CODED  (L)  IF  LESS  THAN  TWO  (2)  POSITION 
LIGHTS  AND  THREE  (3)  FORMATION  LIGHTS  ARE  OPERABLE  ON  EACH 
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Figure  A-1.  Chief  of  Naval  Operations  Instruction  5442.4M  (OPNAVINST 
5442. 4M).  (1992).  Aircraft  material  condition  definitions,  mission-essential 
subsystems  matrices  (MESMS),  and  Mission  Descriptions.  01  July  1992. 
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OPNAVINST  5442. 4M 

17  OCT  1990 


F/A-18A/B/C/D  (oont) 


7. 

8. 
9. 

10. 

11. 


SIDE  OF  AIRCRAFT.  OF  ELECTROMAGNETIC  INTERFERENCE 

DUE  TO  THE  RELATIONSHIP  OF  ELECTROnA^  ^  pROTEC- 

32J5ST&S.  S?S*SS®  ' WITHIN  LIMITS  SPECIFIED  IN 

EQUIPMENT,  MAGNETIC  AZIMUTH 

DETECTOR.  f BOTH  STANDBY  AND  RESET 

5g^J™8£SliS3£.  ATTITUM  REFERENCE  IHMCMO*. 

vgs-jszn&r 

SEVEN  STRAIN  e^HOMZWTAL  TAIL,  LEFTHORIZONTAL  TAIL,  RIGHT 

HIST  BE  OPERABU. 

*  c  APPLICABLE  PER  REFERENCE  (C) . 
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Figure  A-1.  Chief  of  Naval  Operations  Instruction  5442.4M  (OPNAVINST 
5442. 4M).  (1992).  Aircraft  material  condition  definitions,  mission-essential 
subsystems  matrices  (MESMS),  and  Mission  Descriptions.  01  July  1992. 
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APPENDIX  B 
DATABASE  INTERFACES 


Figure  B-1.  Decision 


Knowledge  Programming  for  Logistics  Analysis  and  Technical 


Evaluation  (DECKPLATE)  user  interface. 
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Figure  B-2.  NATEC  ETS  Local  Assistance  Request  (ELAR)  user  interface  for  database 

query. 
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Figure  B-3.  NATEC  ETS  Local  Assistance  Request  (ELAR)  user  interface  for  new 


request  input. 
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APPENDIX  C 
DATA  COMPILATION 


churn 


NMCM 


AMAG 


12/0 1/2C 


AMAG 


AMAG 


AMAG 


03/D 1/2004 


AMAG 


04/D 1/2004 


VMFA224 


AMAG 


05/D 1/2004 


05/01/2C 


AMAG 


06/D 1/2004 


06/0 1/2C 


AMAG 


07/D 1/2004 


AMAG 


08/D 1/2004 


AMAG 


ll/Dl/2004 


AMAG 


12/D 1/2004 


AMAG 


01/D1/2005 


AMAG 


AMAG 


^JSJxJ 


\  X- 


X]  File  Edit  View  Insert  Format  Data  Statistics  Graph  Options  Window  Help 

□  a  H  #  X  %  B  jo  *  K  D  SI  US  la [un2ir  3 ir  in [[3  U  @  V? 

H  |  m  m  m  |  m  a  |  1 1  uii  -  ijn  mi  gg;  tm  |  E  B  E I  a  g.  ?«  |no  Active  Link  3 


AMAG  09/01/2004 
AMAG  10/D 1/2004 


x'VSv  X\Xy  XXtsk  XX.ik  ^'Ss.  XVtCv  '^.K-V  *'&Nxiv 


'mT 


3 

Date 

ll/Dl/2003| 
12/D 1/2003 


01/0 1/2004 
02/D 1/2004 


NMCMU  CannsPerFH  'MMHperFItH  A799PerFH  MMHperMA 


0.09 

0.25 


0.20 

0.29 


TDhrs  MoExplstQ  MoExp3rdQ  lAircraftHrsIn' 

19.00|  28.0o|  82.00|  3906.58| 


16  17  jtJ 

MTBMA  dateCh 

8.87|ll/Dl/2C — I 


14.1001/01/2C 
5.78  02/D  1/2C 


0.33 


4.26  03/0 1/2C 
4.59  04/D  1/2C 


NA01/01/2C 

NA02/01/2C 


07/D1/2C 

08/D1/2C 


4.64  09/D  1/2C 
4.701 10/D  1/2C 


5.551 1/0 1/2C 
5.95I12/D1/2C 


Figure  C-1.  S-Plus  dataframe  used  for  data  compilation. 
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APPENDIX  D 

PAIRWISE  SCATTERPLOTS 


MMHperMA 

■  ^ 

' 

jfc. 

MoExpIstQ 

r 

f. 

0' 

MoExp2ndQ 

'0': 

% 

H h 

W' 

MoExp3rdQ 

#' 

$ 

& 

Iess21 

% 

it.- 

1 

rimelnSqdnQ25 

V 

$0 

v  • 

Jr 

/ 

J&' 

firSv 

1 

s# 

rimelnSqdnMed 

*  . 

%F" 

0” 

■g$': 

’  ^ 

#  '* 

rimelnSqdnQ75 

40- 

35- 

30- 

25- 

20 

15 


-55 

50 

45 

40 

35 

30 


110 

100 

90- 

80 

70- 

60- 

50: 


0.25 

0.20 

0.15 

0.10 

0.05 

0.00 


30- 

20- 

10 

0 


45 

-35 

25 

15 


55 

50- 

45- 

40- 

35 

30- 


0  2  4  6  8 


303540455055 


0.088.  CIS  1ffl.1ffi.2825 


15  25  35  45 


Figure  D-1.  MMHperMA  and  Personnel  Factors  Pairwise  Scatter  Plots.  Each  squadron’s 
data  is  indicated  with  its  own  color  shading. 
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1.0  1.5  2.0  2.5  3.0 


1.01.52.02.53.03.54.0 


1000000000000006000 


MMHperMA 

i  i  i 

*  . 

1  \ 

* 

1  1  M 

• 

•  • 

• 

.» ••  • 

TEC 

9  9 

9  9 

9 

9  9  • 

9  9  9  • 

•  • 

<%««• 

••  • 

•  ft 

9  9  9 

Loc 

ft  it  ft  ft 

9  9  9  9 

a  a 

ft  9 

9  ft 

9  9  9 

V  ft 

9  9 

ft  a 

9  9 

Deployment 

.Vi"  * •  • 

#M*  ••  a 

«  •  •  •  <«• 

#*<«»•  «. 

ftfrcftMft'MW 

• 

•  • 

• 

1  1  i 

1  \ 

:  i  1 

!  6  1  * 

Flthrs 

«  •• 

$ 

lr  ■ 

*  ; 

J  1 

1  f 

i  ! 

l  .  S 

1  *  *  ? 

»  ‘  * 

•#  '• 

AvgAircraftHrsInSvc 

2.5- 


1.5 

i.o- 


4.0 

3.5 
3.0 
2.5- 
2.0- 

1.5 
1.0 


5000- 

4000 


1000- 


2.0 

■1.8 

1.6 

■1.4 

1.2 

1.0 


1500 

1250 

1000 

750 

■500 

■250 

0 


~T~ 

1.01.21.41.61.82.0 


t — i — i — i — r 
0  25(50(7500 0K23K500 


6  8 


Figure  D-2.  MMHperMA  and  Inventory  and  Operational  Factors  Pairwise  Scatter  Plots. 
Each  squadron’s  data  is  indicated  with  its  own  color  shading. 
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Residuals 


APPENDIX  E 

FULL  MODEL  WITH  NATURAL  LOG  TRANSFORMATION  OF 

RESPONSE  VARIABLE 


111  Ys,t  -  Po  +  PlXhs,t  +  PlX2,s,t  +  PlXXs,t  +  P AX A,s,t  +  PsX5,s,I  + 

PbXb,s,t  PlXl,s,l  PsXS,s,t  +  P<)X 9, s,t  PlOXlO,s,l  PuXU,s,t  ^s,t 


Xh. 
X2„ 
X3,. 
X 4, 
X 5,. 
X6,. 
Xl, 
X8,. 
x0 


x, 


10,5/ 


X, 


11,5/  " 

f5/  = 
£= 
5= 

r= 


=  man-hours  per  maintenance  action,  squadron  s,  month  t 
=  type  equipment  code 
=  first  quartile,  months  experience 
=  third  quartile,  months  experience 
=  average  aircraft  hours  in  service 
=  turnover 
=  location 

=  first  quartile,  months  in  squadron 
=  second  quartile,  months  in  squadron 
=  third  quartile,  months  in  squadron 
=  deployment 
=  flight  hours 
=  residual 

=  number  of  variables 
=  squadron 
=  month 


;tQ  +  MoExp3rdQ  +  AvgAircraftHrsInSvc  +  Turnover  +  Loc  +  TimelnSqdnQ25  +  TimelnSqdnMed  +  TimelnSqdnC 

Figure  E-1.  Full  Model  and  Plot  of  Residuals  vs  Fitted  Values. 
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log(MMHperMA) 


stQ  +  MoExp3rdQ  +  AvgAircraftHrsInSvc  +  Turnover  +  Loc  +  TimelnSqdnQ25  +  TimelnSqdnMed  +  TimelnSqdnC 


Figure  E-2.  Plot  of  Response  vs  Fitted  Values,  Full  Model. 


Linear  Model  * 


Call:  lm( formula  =  log  (MMHperMA)  -  TEC  +  MoExplstQ  +  MoExp3rdQ  + 

AvgAi rc raf tHrs InSvc  +  Turnover  +  Loc  +  TimeInSqdnQ25  +  TimelnSqdnMed  + 

TimeInSqdnQ75  +  Deployment  +  Flthrs  +  MoExp2ndQ,  data  =  A,  na. action  = 
na .exclude ) 

Residuals : 

Min  IQ  Median  3Q  Max 

-0.6554  -0.1302  0.01567  0.1086  0.6526 

Coefficients : 


Value  Std.  Error  t  value  Pr(>|t|) 


( Intercept) 

1 

.2280 

0 

.2453 

5 

.  0060 

0 

.  0000 

TECAMAF 

0 

.0517 

0 

.0708 

0 

.  7304 

0 

.  4660 

TECAMAG 

0 

.1865 

0 

.0816 

2 

.2861 

0 

.  0233 

MoExplstQ 

0 

.0011 

0 

.0071 

0 

.  1620 

0 

.  8715 

MoExp3rdQ 

-0 

.0026 

0 

.0022 

-1 

.  1800 

0 

.2395 

AvgAi rc  raf tHrs InSvc 

0 

.0001 

0 

.0000 

1 

.  7798 

0 

.  0767 

Turnover 

-0 

.2300 

0 

.2191 

-1 

.  0497 

0 

.2952 

Loc 

0 

.2214 

0 

.0389 

5 

.6884 

0 

.  0000 

TimeInSqdnQ25 

0 

.0005 

0 

.0065 

0 

.  0815 

0 

.  9351 

TimelnSqdnMed 

-0 

.0179 

0 

.0057 

-3 

.1402 

0 

.0020 

TimeInSqdnQ75 

-0 

.0090 

0 

.0071 

-1 

.2650 

0 

.2074 

DeploymentUDP 

-o 

.1124 

0 

.0512 

-2 

.  1929 

0 

.  0295 

Dep 1 oyme  nt  CVN 

0 

.1827 

0 

.1099 

1 

.  6634 

0 

.  0978 

Deployment IRAQ 

0 

.0680 

0 

.2092 

0 

.  3249 

0 

.  7456 

Flthrs 

0 

.0000 

0 

.0001 

0 

.2025 

0 

.  8397 

MoExp2ndQ 

0 

.0064 

0 

.0072 

0 

.  8896 

0 

.  3748 

Residual  standard  error:  0.212  on  194  degrees  of  freedom 
Multiple  R-Squared:  0.2791 

F-statistic:  5.007  on  15  and  194  degrees  of  freedom,  the  p-value  is  2.751e-0Q8 
194  observations  deleted  due  to  missing  values 

Analysis  of  Variance  Table 

Response:  log (MMHperMA) 

Terms  added  sequentially  (first  to  last) 


Df 

Sum  of  Sq 

Me  an 

Sq 

F 

Val 

ue 

Pr 

(F) 

TEC 

2 

0  . 

286042 

0 

.143 

021 

3 

.180 

82 

0 

.0437273 

MoExplstQ 

1 

0  . 

342736 

0 

.342 

736 

7 

.  622 

53 

0 

.0063 

176 

MoExp3rdQ 

1 

0. 

050081 

0 

.050 

081 

1 

.113 

81 

0 

.2925 

676 

AvgAi rc  raf tHrs InSvc 

1 

0. 

068887 

0 

.068 

887 

1 

.532 

05 

0 

.2173 

005 

Turnover 

1 

0  . 

010424 

0 

.010 

424 

0 

.231 

83 

0 

.  6307 

150 

Loc 

1 

1 . 

589110 

1 

.589 

110 

35 

.342 

16 

0 

.0000 

000 

TimeInSqdnQ25 

1 

0  . 

007769 

0 

.007 

769 

0 

.172 

78 

0 

.  6781 

147 

TimelnSqdnMed 

1 

0  . 

570940 

0 

.570 

940 

12 

.  697 

84 

0 

.0004 

608 

TimeInSqdnQ75 

1 

0. 

010226 

0 

.010226 

0 

.227 

43 

0 

.6339 

737 

Deployment 

3 

0. 

403538 

0 

.134 

513 

2 

.991 

59 

0 

.0  321 

322 

Flthrs 

1 

0  . 

001517 

0 

.001 

517 

0 

.033 

74 

0 

.8544 

627 

MoExp2ndQ 

1 

0  . 

035585 

0 

.035 

585 

0 

.791 

42 

0 

.3747 

722 

Residuals 

194 

8  . 

722932 

0 

.044 

964 

Figure  E-4.  Full  Model  Summary  and  ANOVA  Table,  S-Plus  Report 
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Residuals 


APPENDIX  F 

STEPWISE  REDUCED  MODEL 


ln  Ys,t  -  A  +  P\X\ ,s,t  +  PlX2,s,t  +  03X3 ,s,t  +  PaX A,s,t  + 
fisX5 ,s,t  £s,t 


Ys ,  =  man-hours  per  maintenance  action,  squadron  s,  month  t 
XX  s  t  =  type  equipment  code 
xisi  =  average  aircraft  hours  in  service 
X3  t  =  location 

X4  s  t  =  months  in  squadron,  median 
X5  s  t  =  deployment  status 
sst  =  residual 
k  =  number  of  variables 
s  =  squadron 
t  =  month 


0.6  0.7  0.8  0.9  1.0  1.1  1.2 


Fitted  :  TEC  +  AvgAircraftHrsInSvc  +  Loc  +  TimelnSqdnMed  +  Deployment 


Figure  F-1.  Reduced  Model  and  Plot  of  Residuals  vs  Fitted  Values 
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Fitted  :  TEC  +  AvgAircraftHrsInSvc  +  Loc  +  TimelnSqdnMed  +  Deployment 


Figure  F-2.  Response  vs  Fitted  Values,  Reduced  Model. 
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Fitted  :  AvgAircraftHrsInSvc  +  TimelnSqdnMed  +  Deployment  +  Org 


Figure  G-2.  Plot  of  Response  vs  Fitted  Values,  Final  Model. 
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